How to get oobabooga/text-generation-webui running on Windows or Linux with LLaMa-30b 4bit mode via GPTQ-for-LLaMa on an RTX 3090 start to finish.
This guide actually works well for linux too. Just don't bother with the powershell envs
-
Download prerequisites
- Download and install miniconda
- (Windows Only) Download and install Visual Studio 2019 Build Tools
- Click on the latest BuildTools link, Select Desktop Environment with C++ when installing)
-
(Windows Only) Open the Conda Powershell.