Qwen3.8-27B on RTX 5090 — 256K, vision, turbo4 KV, fused MMA, MTP

Live stack: Unsloth UD-Q6_K + mmproj-F16, Madreag turbo3-cuda sync/2026-08, native 262144 context, turbo4/turbo4 KV (main + draft), fused MMA + GDN, native MTP n=3.

Hardware this was measured on: RTX 5090 32 GB, CUDA 13.3, SM120.

Perf (RTX 5090, this exact command)

Production soak on the fused-MMA + GDN + MTP stack above. Same GGUF, 256K ctx reserved, one slot. VRAM stays flat after load.

Headline


VRAM at 262K reserved	29090 / 32607 MiB (~3040 free). Idle fused-MMA ~29.0 GiB
vs previous `q8_0/turbo4`	31418 used / 710 free — this stack frees ~3 GB
Lifetime decode	~87–96 t/s average across millions of gen tokens
Lifetime MTP accept	~62–69% (mean draft length ~2.3–2.9 of n=3)
Deepest fill	197182 tokens (`truncated=0`). Configured max 262144
GPU	93–97% @ 63–77°C under load

Decode vs context depth (live traffic, not llama-bench)

Depth	Decode	MTP accept
short / code, 256 tok	132 then 150 t/s	67–81%
~8k	122–131 t/s	80–92%
~18k	93 t/s (543 gen)	45% (mean len 2.34)
~33k	130 t/s (8k–14k gen)	~80%
~108–113k	66–98 t/s typical; 94 t/s / 81% on a 288-tok turn	55–81%
~143k	63–78 t/s (2k–7k gen)	—
~157k	64 t/s	—
162–168k	57–85 t/s	48–84%
191k	52 t/s (186k prefill + 4.3k gen)	46% (mean len 2.38)

Copy-heavy / structured turns sit at the high end of accept. Mixed chat sits lower. Accept falling with depth is expected.

Prefill


Cached / tiny prompt (30 tok @ ~18k depth)	~250 t/s
1.3k tok	~2400 t/s
Long re-prefill ~10–16k	~3200–3300 t/s
Full 167k–186k re-prefill	1285–1749 t/s

Short-prompt A/B (same UD-Q6_K GGUF, 2026-08-19)

	`q8_0/turbo4` MTP	this `turbo4/turbo4` fused+GDN MTP
Code 256 tok	116 t/s, 56% accept	132 then 150 t/s, 67–81%
VRAM at 262K	31418 / 710 free	28984 / 3144 free

One 41h soak snapshot: 3.13M gen tokens in 36009 s → 87 t/s avg, MTP 61.8%, n_tokens_max 197182, VRAM still 29090 / 3038 free, no OOM / CUDA / crash, all finished turns truncated=0.

1. Build

git clone https://github.com/Madreag/turbo3-cuda.git
cd turbo3-cuda
git checkout sync/2026-08

cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120a \
  -DCMAKE_CUDA_COMPILER=/opt/cuda/bin/nvcc
cmake --build build --target llama-server -j

5090 = SM120. Use 120a on CUDA 13.x. Other GPUs: 86 (3090), 89 (4090), 120 (5090 on older CUDA).

Tested commit: adcf720f3 (origin/sync/2026-08).

2. Weights

huggingface-cli download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-UD-Q6_K.gguf mmproj-F16.gguf \
  --local-dir ~/models/qwen38

~21 GB weights + ~0.9 GB vision projector. Fits a 32 GB 5090 with 256K turbo4 KV and headroom.

Pack: unsloth/Qwen3.8-27B-GGUF

3. Env (this is the gotcha)

export LD_LIBRARY_PATH="$PWD/build/bin${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
export GGML_TURBO_MMA_FUSED=1
export TURBO_NORM_ALPHA_V=1.00
export TURBO4_NORM_ALPHA_V=1.00

Qwen3.8 wants 1.00, not the 3.5-era 1.10 (that one ~2.3× KLD).
Fused MMA only fires if K and V are both turbo4 on the main model and the MTP draft (-ctk/-ctv and -ctkd/-ctvd).

4. Serve

./build/bin/llama-server \
  -m ~/models/qwen38/Qwen3.8-27B-UD-Q6_K.gguf \
  --mmproj ~/models/qwen38/mmproj-F16.gguf \
  -np 1 --ctx-size 262144 \
  -ctk turbo4 -ctv turbo4 -ctkd turbo4 -ctvd turbo4 \
  --flash-attn on --load-mode none \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  -ngl 99 -b 512 -ub 512 \
  --no-context-shift --jinja \
  --reasoning on --reasoning-budget 16000 \
  --reasoning-budget-message "You've done enough thinking. Give your final answer now. Do not start a new thinking block." \
  --port 8081 --host 0.0.0.0 --metrics --cache-ram 32768

OpenAI-compatible API on http://127.0.0.1:8081.

Why those flags

Flag	Why
`-ctk turbo4 -ctv turbo4` + the same for draft	Required for fused MMA; ~3 GB more VRAM free vs `q8_0/turbo4` at 256K
`--spec-type draft-mtp --spec-draft-n-max 3`	Native multi-token prediction. Do not stack ngram-mod on this fused tree
`--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0`	Qwen thinking-mode defaults
`--ctx-size 262144 --no-context-shift`	Native 256K window, no sliding
`--reasoning on --reasoning-budget 16000`	Thinking, then injects the stop-thinking message
`--mmproj mmproj-F16.gguf`	Vision
`--cache-ram 32768`	32 GB prompt cache in RAM
`-np 1 -ngl 99 -b 512 -ub 512`	One slot, fully offloaded, batch 512

Optional systemd user unit

[Unit]
Description=Qwen3.8-27B Unsloth UD-Q6_K + mmproj-F16, turbo4/turbo4, 256K, MTP n=3
After=network.target

[Service]
Type=simple
Environment=PATH=/opt/cuda/bin:/usr/local/bin:/usr/bin
Environment=LD_LIBRARY_PATH=%h/inference/turbo3-cuda/build/bin
Environment=GGML_TURBO_MMA_FUSED=1
Environment=TURBO_NORM_ALPHA_V=1.00
Environment=TURBO4_NORM_ALPHA_V=1.00
ExecStart=%h/inference/turbo3-cuda/build/bin/llama-server \
  -m %h/models/qwen38/Qwen3.8-27B-UD-Q6_K.gguf \
  --mmproj %h/models/qwen38/mmproj-F16.gguf \
  -np 1 --ctx-size 262144 \
  -ctk turbo4 -ctv turbo4 -ctkd turbo4 -ctvd turbo4 \
  --flash-attn on --load-mode none \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  -ngl 99 -b 512 -ub 512 \
  --no-context-shift --jinja \
  --reasoning on --reasoning-budget 16000 \
  --reasoning-budget-message "You've done enough thinking. Give your final answer now. Do not start a new thinking block." \
  --port 8081 --host 0.0.0.0 --metrics --cache-ram 32768
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Adjust the binary path to wherever you cloned the fork. Then:

systemctl --user daemon-reload
systemctl --user enable --now llama-qwen38.service

Credits

Weights: Qwen/Qwen3.8-27B
GGUF: unsloth/Qwen3.8-27B-GGUF (Dynamic V3 UD-Q6_K)
Server: Madreag/turbo3-cuda (sync/2026-08) — TurboQuant KV + fused MMA/GDN on llama.cpp

unknownliviu/qwen38-27b-rtx5090.md

Select an option

No results found