GPT-OSS on AMD Radeon 890M (llama.cpp)

gpt-oss-amd-radeon-gpu-890M.md

GPT-OSS on AMD Radeon 890M (llama.cpp)

Local inference for OpenAI gpt-oss-20b via llama.cpp and Vulkan.

Why gpt-oss-20b

Model	Fits this GPU?	Notes
gpt-oss-20b	✅ Yes	MXFP4 GGUF ~11.3 GiB; full GPU offload on 890M
gpt-oss-120b	❌ No	Needs ~80 GB VRAM (or heavy CPU offload + 64 GB+ RAM)

Weights: ggml-org/gpt-oss-20b-GGUF → gpt-oss-20b-mxfp4.gguf (~12 GB on disk)

System

• GPU: AMD Radeon 890M (gfx1150, ~13.5 GB UMA)
• Backend: Vulkan (ROCm untested for gpt-oss; Vulkan is stable here)
• Build: llama.cpp 6e14286, -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release

Quick start

# One-shot prompt
./run-gpt-oss.sh "What is 17+25? One short sentence."

# OpenAI-compatible server → http://127.0.0.1:8080
./run-gpt-oss-server.sh

Manual CLI

cd llama.cpp
./build/bin/llama-cli \
  -m ../models/gpt-oss-20b-mxfp4.gguf \
  -ngl 99 -fa 1 -c 8192 -b 512 -ub 512 \
  --jinja --single-turn --simple-io \
  -p "Your prompt here" -n 256

If you run out of VRAM

Reduce context or offload MoE layers to CPU:

CTX=4096 ./run-gpt-oss.sh "prompt"

# or (llama-server / llama-cli)
--n-cpu-moe 8   # try 8, 12, 16 until it loads

Performance (Radeon 890M, Vulkan)

Measured 2026-06-14 on this machine.

Test	Throughput
Token generation (tg128, llama-bench)	27.1 t/s
Interactive generation (8192 ctx)	23–26 t/s
Prompt processing (short prompt)	128–162 t/s

Settings: -ngl 99 -fa 1 -c 8192 -b 512 -ub 512, model fully on GPU.

Community reference (same GPU, more RAM): ~27 t/s generation on 890M Vulkan (llama.cpp #15396).

Rebuild

cd llama.cpp
git pull
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON
cmake --build build -j"$(nproc)"

Download model

python3 -m venv .venv && . .venv/bin/activate
pip install huggingface_hub
huggingface-cli download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models

Benchmark

cd llama.cpp
./build/bin/llama-bench \
  -m ../models/gpt-oss-20b-mxfp4.gguf \
  -ngl 99 -t 1 -fa 1 -p 0 -n 128 -b 2048 -ub 512

---

Bonsai 1.7B on AMD iGPU (Ryzen 7 5800H / Cezanne gfx90c)

Measured 2026-07-31 with PrismML Bonsai 1.7B Q1_0 via bonsai-ollama + Prism llama-server (prism-b9596-9fcaed7).

System

CPU	AMD Ryzen 7 5800H (8C/16T)
GPU	Radeon Vega (Cezanne, gfx90c, RADV RENOIR, ~6.5 GB UMA)
GGUF	Bonsai-1.7B-Q1_0.gguf (~248 MiB)
Prism CPU/Vulkan	llama-prism-b8846-d104cf1-bin-ubuntu-x64 (Vulkan backend in bundle)
Prism ROCm	llama-prism-b9596-9fcaed7-bin-ubuntu-rocm-7.2-x64

Results (bin/bench_llama_tokens --runs 3, -ngl 99 -np 1 -fit off)

Backend	Decode (tok/s)	Prefill (tok/s)	GPU offload	Status
Vulkan (-ngl 99)	60.9 (σ 0.2)	59.1 (σ 0.8)	29/29 layers	✅
CPU (-ngl 0)	60.3 (σ 0.4)	45.6 (σ 6.1)	none	✅
ROCm 7.2 (-ngl 99)	—	—	—	❌ cudaMalloc / HSA assert on gfx90c

Vulkan log: offloaded 29/29 layers to GPU on Vulkan0 (AMD Radeon Graphics (RADV RENOIR)).

ROCm fails on this APU (not in Prism/aigdat prebuilt GPU target lists). Use Vulkan for GPU offload on Cezanne; use ROCm on gfx103X+ / gfx110X / gfx1150+.

Reproduce (with timeouts — avoids hangs)

git clone https://github.com/eSlider/bonsai-ollama.git
cd bonsai-ollama
./bin/setup.sh

# Vulkan GPU (recommended on gfx90c)
export BONSAI_PRISM_LIB_DIR="$PWD/vendor/prism-llama/llama-prism-b8846-d104cf1"
export BONSAI_LLAMA_EXTRA_ARGS='-ngl 99 -np 1 -fit off'
BONSAI_BENCH_STARTUP_TIMEOUT=180 BONSAI_BENCH_TIMEOUT=300 ./bin/bench_bonsai_prism.sh
# ^ script name is generic; point BONSAI_PRISM_LIB_DIR at Vulkan or ROCm extract

# Or Prism ROCm tarball (setup):
BONSAI_SETUP_PRISM_VARIANT=rocm ./bin/setup.sh
BONSAI_PRISM_LIB_DIR="$PWD/vendor/prism-llama/llama-prism-b9596-9fcaed7-rocm" \
  BONSAI_BENCH_STARTUP_TIMEOUT=600 ./bin/bench_bonsai_prism.sh

Helper script: bin/bench_bonsai_prism.sh — env BONSAI_BENCH_STARTUP_TIMEOUT (default 600s) and BONSAI_BENCH_TIMEOUT (default 300s) prevent indefinite freezes during first HIP compile or bad GPU state.

llama.cpp ROCm smoke bench (Qwen3-0.6B)

For llamacpp-rocm local builds (TheRock gfx90c):

BENCH_TIMEOUT=300 ./utils/bench-gpu.sh   # wraps llama-bench with timeout

---

OpenCode CLI

OpenCode is configured to use the local llama-server via an OpenAI-compatible adapter.

Install (once)

curl -fsSL https://opencode.ai/install | bash
cd /home/devops/projects/tts/.opencode && npm install

Config files:

• opencode.json (project)
• ~/.config/opencode/opencode.json (user)

Run

Terminal 1 — model server (16384 ctx, 1 slot for agent prompts):

CTX=16384 NP=1 ./run-gpt-oss-server.sh

Terminal 2 — OpenCode (starts servers automatically if needed):

./run-opencode.sh "Explain what run-gpt-oss-server.sh does."

Interactive TUI:

./run-opencode.sh

Inside OpenCode: /models → select llama.cpp/gpt-oss-20b-mxfp4.gguf.

Notes

• OpenCode agent system prompts are ~6k tokens; use ctx ≥ 16384 or you'll get Context size has been exceeded.
• Requires @ai-sdk/openai-compatible in ~/.opencode and .opencode/node_modules.
• Model ID must match llama-server: gpt-oss-20b-mxfp4.gguf.