Platform: Intel Core Ultra 7 270K Plus (24C/24T), 96GB DDR5, RTX 4090 24GB.
Model: unsloth/Qwen3.8-Flash-Next-GGUF, UD-IQ3_XXS (~78GB) + mmproj-F16 (~900MB, vision enabled).
llama.cpp mainline (ghcr.io/ggml-org/llama.cpp:full-cuda13). Final config as of 2026-08-29.
Docker Compose (ready to run): see attached docker-compose.qwen3.8-flash-next.yml.
llama-server \
--model Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
--mmproj mmproj-F16.gguf \
--ctx-size 120000 \
--fit on --fit-target 256 \
-t 20 -tb 24 \
-b 6144 -ub 6144 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \
--flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
--reasoning-budget 4000 \
--reasoning-budget-message "... reasoning budget exceeded, need to answer.\n" \
--chat-template-kwargs '{"reasoning_effort": "medium"}' \
--reasoning-preserve \
--image-max-tokens 4000 --image-min-tokens 1024 \
--cont-batching --metrics -np 1| Flag | Reason (measured on this box) |
|---|---|
--ctx-size 120000 |
smaller KV alloc → --fit offloads more weights → decode +8% (22.2 vs 20.4 t/s @ 90K). 110K input still 19.5 t/s. Cost: sessions >120K get truncated (client-side compaction handles it) |
--fit on --fit-target 256 |
auto offload beats manual -ncmoe (84K input: 19.5 vs 16.2 t/s); 512→256 frees room for more weights (+0.3 t/s, no OOM at -ub 6144). Don't go below 256 |
-t 20 -tb 24 |
prefill: 24T=836, 20T=848 t/s (leave 4 cores for system); decode-batch uses full 24 |
-b/-ub 6144 |
prefill sweet spot: 2048=762, 4096=850, 6144=864, 8192=828 t/s. Keep -b == -ub or 8192 becomes a 6144+2048 tail batch and slows down |
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 |
unsloth-recommended sampling for thinking mode |
--flash-attn on --cache-type-k/v q8_0 |
saves ~1.5GB VRAM vs F16 KV. q4_0 KV was tried: dequant overhead cancels the gain (28.9 vs 30.1 t/s @32K), so q8_0 stays |
--reasoning-budget 4000 + reasoning_effort: medium |
default xhigh wastes tokens; high effort degrades quality on this model (self-doubt loops) |
--reasoning-preserve |
pass reasoning history back to model across turns |
--mmproj mmproj-F16.gguf |
vision enabled by default; costs ~20-30K ctx vs text-only |
-np 1 |
single slot → max per-slot context |
| Input | Prefill t/s | Decode t/s | Note |
|---|---|---|---|
| 6K | ~1360 | ~30 | warm |
| 90K | ~905 | ~22 | typical agent/coding session |
| 110K | ~863 | ~19.5 | 92% of ctx, near-full |
Decode tracks absolute KV depth: 6K→30, 90K→22, 110K→~19.5 t/s. ctx allocation itself doesn't change speed — the gain from 120K comes purely from freed VRAM going to weight offload.
- Request over ctx → HTTP 400 with
n_prompt_tokens/n_ctxin the response. At ctx 120K plan client-side compaction before the limit. --ctx-size 131072OOMs with mmproj (auto-reduced to 130560); max stable with mmproj is ~150K, text-only ~170K.- Mainline llama.cpp has no KVarN KV quant; q8_0 + flash-attn is the practical best for this model.
- Model ships an MTP head but llama.cpp doesn't support it yet (
--spec-type draft-mtpOOMs). Watch for upstream updates — could be a big decode win. - Docker image note:
full-cuda13entrypoint is a tool dispatcher, pass--serverexplicitly. - 96GB RAM is enough to hold the ~78GB weights fully resident with swap unused; with less RAM expect cold-start penalties on large prefills.
How would you get this work with a 4090 but 64GB of RAM?
Also, is it so much better over Qwen3.8-27b ?