WARNING: This is only for headless Framework Desktop and other AI MAX 395+ 128GB machines. I tried this on my Asus ROG Z13 with KDE running and it crashed my system hard. If you're using LLMs on a machine with a desktop environment, consider running llama.cpp server with the Vulkan backend instead of this.
First you have to set up your Framework Desktop to allow a large amount of GTT memory.
This was tested with the following modprobe.conf settings:
# Maximize GTT for LLM usage on 128GB UMA system
options amdgpu gttsize=120000
options ttm pages_limit=31457280
options ttm page_pool_size=15728640
However, some of these options are outdated.
I used the VLLM recipe here as a starting point:
https://recipes.vllm.ai/Google/gemma-4-26B-A4B-it
docker run -d --restart=unless-stopped \
--device=/dev/kfd --device=/dev/dri \
--security-opt seccomp=unconfined --group-add video \
--privileged --ipc=host -p 8013:8013 \
--name gemma4 \
-v vllm-cache:/root/.cache/huggingface \
vllm/vllm-openai-rocm:nightly \
google/gemma-4-26B-A4B-it \
--host 0.0.0.0 --port 8013 \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--max-model-len -1 \
--async-scheduling \
--chat-template vllm/examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
For some reason, the ROCM image has a different working directory so the --chat-template argument is a bit different from the recipe.
I then used this setup with pi.dev using a models.json like so:
{
"providers": {
"vllm": {
"baseUrl": "http://192.168.9.20:8013/v1",
"api": "openai-completions",
"apiKey": "sk-local",
"models": [
{
"id": "google/gemma-4-26B-A4B-it",
"input": ["text", "image"],
"reasoning": true,
"contextWindow": 262144,
"maxTokens": 8192
}
]
}
}
}