llama.cpp server AMD Radeon RX 7900 XTX perfect fit

README.md

This llama-server setup is specifically tuned to my AMD Radeon RX 7900 XTX for running gemma 4 26B A4B quantized by unsloth.

I've set it up to ensure it's stable, preferring as much practical quality as possible despite the VRAM limits.

This utilizes 99% of the VRAM on my setup so there's no room for improvement.

I get somewhere between 100-120 tokens/second token generation speeds with a single user.

start_llama_server.sh

#!/bin/sh

docker run -d --restart=unless-stopped \
    --device /dev/kfd --device /dev/dri \
    --security-opt seccomp=unconfined --group-add video \
    --privileged --ipc=host \
    -v huggingfacehub:/root/.cache/huggingface/hub \
    --network=host \
    -e LLAMA_ARG_HOST=0.0.0.0 \
    --name llama-server \
    ghcr.io/ggml-org/llama.cpp:server-vulkan \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    -ngl 999 \
    -kvu -ctk q5_1 -ctv q5_1 \
    -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q5_K_XL \
    --no-mmap \
    --cache-prompt \
    --port 8013 --reuse-port