A coding agent that runs entirely on your own machine. No API keys, no cloud, works offline.
pi is the agent you talk to. It runs against a local llama.cpp server hosting Google Gemma 4 (the unsloth builds).
Pick a model by the memory you have -- one model, one full-context config, nothing to tune. The indicator differs by platform:
| Model | unsloth repo | Download | macOS (unified memory) | Linux (GPU VRAM) | Context (maxed) |
|---|---|---|---|---|---|
| Gemma 4 E4B | unsloth/gemma-4-E4B-it-GGUF |
5.13 GB | 16 GB | 12 GB | 131072 (128k) |
| Gemma 4 12B | unsloth/gemma-4-12b-it-GGUF |
7.37 GB | 32 GB | 16 GB | 262144 (256k) |
- macOS (Apple Silicon): go by total unified memory -- it is shared with the OS and your apps, so the bar sits higher. 16 GB -> E4B, 32 GB -> 12B.
- Linux (discrete GPU): go by the graphics card's dedicated VRAM. 12 GB -> E4B, 16 GB -> 12B.
Each alias runs its model at full context, in a single conversation slot, with
--cache-ram 0. pi is a single-threaded, synchronous, one-conversation harness, so
extra slots and a cross-chat KV cache would only spend memory on things pi never uses.
Dropping them keeps memory pressure low enough to keep using the machine for normal
work.
This gives you the llama binary; you start the server with llama serve. Confirm
the install with llama version.
macOS -- Homebrew:
brew install llama.cppLinux (any distro) -- grab a prebuilt binary with installama.sh.
This script auto-detects your CPU and GPU (CUDA / ROCm / Vulkan) and drops llama
into ~/.local/bin (it tells you if that is not on your PATH):
curl -fsSL https://angt.github.io/installama.sh | shAdd an alias to start the server. It also saves the server's output to a timestamped
log file, which helps when something goes wrong. Put this in your shell config --
~/.zshrc on macOS, ~/.bashrc on Linux:
# Gemma 4 E4B @ 128k full context for pi -- 16 GB Mac / 12 GB Linux VRAM (f16 KV, flash-attn, :8080, vision on, pi retain template, single slot, --cache-ram 0)
alias gemma4-e4b='mkdir -p ~/.llama-logs && llama serve -hf unsloth/gemma-4-E4B-it-GGUF:UD-Q4_K_XL -c 131072 -fa 1 --jinja --chat-template-file ~/.llama-templates/gemma-4-e4b-pi.jinja --parallel 1 --cache-ram 0 --temp 1.0 --top-p 0.95 --top-k 64 --min-p 0 2>&1 | tee ~/.llama-logs/gemma4-e4b-$(date +%Y%m%d-%H%M%S).log'
# Gemma 4 12B @ 256k full context for pi -- 32 GB Mac / 16 GB Linux VRAM (single slot, --cache-ram 0 for low memory pressure, + --reasoning on)
alias gemma4-12b='mkdir -p ~/.llama-logs && llama serve -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL -c 262144 -fa 1 --jinja --chat-template-file ~/.llama-templates/gemma-4-12b-pi.jinja --parallel 1 --cache-ram 0 --temp 1.0 --top-p 0.95 --top-k 64 --min-p 0 --reasoning on 2>&1 | tee ~/.llama-logs/gemma4-12b-$(date +%Y%m%d-%H%M%S).log'Both aliases point at a custom chat template with --chat-template-file. Put the
matching file in place before the first launch, or the server will not start. Each
model needs its own -- they share the same pi communication flow but sit on separate
Gemma 4 bases, so the files are not interchangeable:
mkdir -p ~/.llama-templates
curl -fsSL https://gist.githubusercontent.com/joematthews/84b55f8bb1cc3e3adfce7ba7ec01275c/raw/gemma-4-e4b-pi.jinja -o ~/.llama-templates/gemma-4-e4b-pi.jinja # gemma4-e4b
curl -fsSL https://gist.githubusercontent.com/joematthews/9be31aee86fb1e1c195ba14289b70c41/raw/gemma-4-12b-pi.jinja -o ~/.llama-templates/gemma-4-12b-pi.jinja # gemma4-12bWhy it matters. Gemma 4's stock template trims the model's reasoning from history on each new turn. That rewrites the prompt prefix every turn, so llama.cpp cannot reuse its KV cache and reprocesses the whole conversation -- slow, and worse the longer the session runs.
Both aliases point at a pi retain build (gemma-4-e4b-pi.jinja for E4B,
gemma-4-12b-pi.jinja for the 12B). Each keeps prior reasoning in history and holds
the rendered prefix byte-stable across turns, so llama.cpp reuses its prefix cache
instead of reprocessing every turn. Each also carries its base's tool-calling fixes
(string-or-JSON tool arguments, null-safe rendering), which is what pi relies on to
call tools reliably. The two files apply the same pi changes onto their own Gemma 4
base -- the 12B base carries a generation-prompt marker the E4B base does not, which
is why they are ported separately rather than shared byte-for-byte.
The retain approach originates in the community fork discussed in llama.cpp #21912, where vevi33 shared the template change and aldehir identified the underlying cause. Credit to both. The pi build layers pi-specific tool-calling and prefix-stability tuning on top of that base.
Then start it:
source ~/.zshrc # or ~/.bashrc
gemma4-e4b # first run downloads the model (~5 GB), then serves on :8080Note
The first run downloads the model automatically, which takes a while. You may see no output for a few minutes -- that is the download. The server starts once it finishes. Later starts are instant.
| flag | what it does |
|---|---|
--cache-ram 0 |
No cross-chat KV cache. Keeps memory pressure low so the machine stays usable for normal work. Trade-off: switching chats reprocesses pi's injected prefix (KV rehydration past the first checkpoints) instead of restoring it from RAM. |
--parallel 1 |
One server slot. Left on auto this build takes 4, with a unified KV buffer and a larger sliding-window cache to cover them; pi runs one conversation, so the rest go unused. Each conversation gets the full -c either way. |
-c |
The context window in tokens, maxed per model: 131072 (128k) on E4B, 262144 (256k) on the 12B. |
-fa 1 |
Flash attention -- faster, and uses less memory. |
--temp 1.0 --top-p 0.95 --top-k 64 --min-p 0 |
Gemma 4's recommended sampling settings. |
--reasoning on |
Enables reasoning. Needed only by the 12B, which has reasoning off by default. |
--chat-template-file ...gemma-4-*-pi.jinja |
The pi retain template (one per model: E4B and 12B) -- keeps reasoning in history so llama.cpp reuses its prefix cache instead of reprocessing every turn (see above). |
Gemma 4 can read images, which adds about 1.2 GB of memory; add --no-mmproj to the
alias if you would rather run text-only.
You do not need to set anything for thinking -- Gemma 4 handles it, and pi controls it. Unlike some models, turning thinking on here does not slow things down.
# install pi
curl -fsSL https://pi.dev/install.sh | sh
# install extension packages
pi install npm:pi-llama-cpp # connects pi to the llama.cpp server
pi install npm:pi-smart-fetch # lets the agent read web pages
pi install npm:pi-smart-web-search # lets the agent search the webpi-llama-cpp-- finds your local server automatically.pi-smart-fetch-- lets the agent read web pages.pi-smart-web-search-- lets it search the web, no API key needed.
Together they let a small model look things up instead of guessing.
# terminal 1: start the model server
source ~/.zshrc && gemma4-e4b
# terminal 2: open your project and start coding
cd ~/your-project
piThat is the whole setup -- a private coding agent that runs on a modest laptop.
Give it a spin. These all need up-to-date info from the web, which is exactly where a small local model needs a hand:
What's the latest version of Node.js, and what's new in it?
Compare Bun and Deno for a new TypeScript API in 2026.
Scaffold a minimal Vite + React + TypeScript app in ./demo, then explain the structure.
Read package.json and tell me which dependencies are out of date.
Find the current recommended way to set up GitHub Actions for a Node project, then write the workflow file.
Watch what it does: it searches, opens the most useful results, reads them, and answers from what it read -- instead of guessing from old training data.