The following steps can be run automatically for you here: https://rosmur.github.io/local-llm-setup/

A. LLM Running Software (Inference Engine)

curl -LsSf https://llama.app/install.sh | sh

B. LLMs (Model Weights Download)

	Model	Download	Recommended Min RAM	Notes
1	Gemma 4 E2B	~3 GB	8GB	The small, fast one. Works on modest machines. A reasonable first choice if you're unsure.
1	Gemma 4 E4B	~4.6 GB	12GB	The small, fast one. Works on modest machines. A reasonable first choice if you're unsure.
2	Gemma 4 26B-A4B (QAT)	~15 GB	24GB	Much more capable, but only activates a small slice of itself per word, so it stays fast.
3	Qwen3.6 35B-A3B	~20 GB	32 GB	Same idea, different family. Stronger at code.

Each of the above have several flavors when you search HF. Recommend

Using quantized versions a.k.a quants. 4-bit (Q4_K_M) quantization is a happy medium that preserves accuracy reasonably while being small.
Use quants from either official sources (google, Qwen etc.) or from unsloth/bartwoski (the latter two are independent developers but have been in the ecosystem from the start in 2023 and are reliable, trusted sources)

To get and use the model:

Click on the quantization of your choice.
Click the "Use this model button" towards the top right in the sidebar that opens.
Copy the llama serve... command (second line)
Paste into a new terminal tab and hit enter
The model will start downloading and will be served (will be live) after the download completes
To use in the future, simply run the same command again (it will not re-download as the model is saved to your hard disk. Defaults to the cache folder)

curl -fsSL https://pi.dev/install.sh | sh

(Copied from llama.app web page)

llama serve

pi install git:github.com/huggingface/pi-llama

To start pi, simply type:

pi

in the terminal. Recommend changing directory (cd path/to/folder) to a specific project or new folder and then starting pi