This document refreshes the curated catalog in 05 against what actually exists and is downloadable on 2026-06-23, and it folds in the speculative-decoding angle that docs 15 and 16 made load-bearing. Doc 05 is still the canonical catalog for column semantics, the master table, and the gateway wiring; this doc is the dated overlay that corrects the model names, re-sizes the picks against the same ~22 GB usable budget, and adds one new column to every large pick: the draft model or built-in MTP head it speculates against.
Two things changed since 05 and 01 were written. First, the model landscape moved: several names the early landscape doc guessed at (Qwen3.6/3.5, DeepSeek-V4, Gemma 4, Llama 4, GLM-5.2, Kimi K2.7, Phi-5) resolved to real releases that look nothing like the guesses, and some of those releases are far too big to serve here. Second, docs 15 and 16 committed the box to an in-process llama.cpp engine whose single largest dense lever is speculative decoding, which means every dense pick now needs a vocab-matched draft and every MoE pick needs to know whether it carries a usable MTP head.
Read 05 for the budget contract and column meanings, 15 section 3 for why speculation is worth 1.8-2.9x on dense code, and 04 for the quant quality table this doc's footprints assume.
- Verification and corrections: what the early landscape got wrong
- Tier 1: general chat / daily driver (30B-class MoE)
- Tier 2: coding
- Tier 3: reasoning
- Tier 4: peak dense that fits
- Tier 5: multimodal / vision
- Tier 6: small/fast and the draft models for speculative decoding
- Tier 7: embeddings
- Does NOT fit 24 GB: the ceiling the box is measured against
- Quant recommendations (tying to doc 15)
- Known gotchas
- Bottom-line default picks
- Decision record
- Cross-references
The landscape doc (01) was written ahead of the releases and named families speculatively. This section reconciles those guesses against what shipped, because three of them changed the catalog and four of them are now "too big, do not recommend" entries that belong in section 9. Benchmark figures attributed to vendors below are vendor-reported and treated as directional, not as box-measured truth.
| Early landscape name | What actually shipped | Fits 24 GB? | Source |
|---|---|---|---|
| Qwen3.6 / Qwen3.5 | Qwen3.6: 27B dense + 35B-A3B MoE, Apache-2.0, shipped 2026-04-16. This is the current OPEN Qwen generation. Qwen3.7 exists but is API-only / closed weights. | Yes (both Qwen3.6 sizes) | huggingface.co/Qwen/Qwen3.6-35B-A3B, qwen.ai/blog?id=qwen3.6-27b |
| DeepSeek-V4 | Real: V4-Pro 1.6T total / 49B active, V4-Flash 284B total / 13B active, MIT, preview 2026-04-24. | No (both far too big) | api-docs.deepseek.com/news/news260424 |
| Gemma 4 | Real: E2B / E4B / 26B-MoE (~3.8B active) / 31B dense, Apache-2.0, 2026-04-02, multimodal. | Yes (E-series, 26B-MoE, 31B dense tight) | blog.google/innovation-and-ai/technology/developers-tools/gemma-4 |
| Llama 4 | Real but dated: Scout / Maverick, EU-restricted license. Even Scout (109B) does not fit. | No (Scout 109B minimum) | huggingface.co/blog/llama4-release |
| Mistral Large 3 / Small 4 | Large 3 is 675B (too big). Small 4 is 24B dense, multimodal, Apache-2.0, 2026-03-16. | Small 4 yes; Large 3 no | mistral.ai/news/mistral-3 |
| GLM-5.2 | Real: 744B total / 40B active, MIT, 2026-06-13, vendor-reported #1 open on the Artificial Analysis index, no vision. | No (744B) | latent.space (GLM-5.2 coverage, vendor-reported) |
| Kimi K2.7 | Name is wrong as a general model. The June release is Kimi K2.7-Code (coding-specialized, 1T total / 32B active). The general flagship is K2.6. Both ~340 GB+. | No (both impossible on 24 GB) | nerova.ai (Kimi K2.7-Code coverage) |
| Phi-5 | No Phi-5 exists. Current is Phi-4-reasoning-vision-15B, MIT, 2026-03-04. | Yes | huggingface.co/microsoft/Phi-4-reasoning-vision-15B |
| gpt-oss | gpt-oss-20b: 21B total / 3.6B active, MXFP4, Apache-2.0 (fits 24 GB easily). gpt-oss-120b needs ~80 GB (does not fit). | 20b yes; 120b no | openai.com/index/introducing-gpt-oss |
Three corrections add real picks: Qwen3.6 (both sizes) becomes the spine of the catalog, Gemma 4 26B-MoE and 31B dense replace the placeholder Gemma 4 entries with real architectures, and gpt-oss-20b enters as a genuinely small high-quality MoE. Mistral Small 4 (24B dense, multimodal) is a real Apache-2.0 dense fallback.
Four corrections move names into the "ceiling" section: DeepSeek-V4 (both variants), GLM-5.2, Kimi K2.6 / K2.7-Code, Mistral Large 3, Llama 4 Scout/Maverick, and gpt-oss-120b are all too big for 24 GB and are documented in section 9 as the reference ceiling, not as serveable models.
One correction is a non-event for fit but matters for naming: there is no Phi-5, and the current Phi is a 15B reasoning-vision model, so the catalog tracks Phi-4-reasoning-vision-15B, not a guessed Phi-5.
Licensing matters because this box may serve commercial work, so the catalog tracks it per pick.
| Pick | License | Commercial use | Note |
|---|---|---|---|
| Qwen3.6 (27B / 35B-A3B) | Apache-2.0 | yes | clean, no field restrictions |
| Qwen3-Coder-30B-A3B | Apache-2.0 | yes | clean |
| gpt-oss-20b | Apache-2.0 | yes | clean |
| Gemma 4 (all sizes) | Apache-2.0 | yes | per the Gemma 4 announcement |
| Mistral Small 4 / Devstral | Apache-2.0 | yes | clean |
| Phi-4-reasoning-vision-15B | MIT | yes | clean |
| Qwen3-Embedding family | Apache-2.0 | yes | clean |
| BGE-M3 | MIT | yes | clean, the strictest-license embedding fallback |
| Llama 4 (ceiling) | Llama 4 Community + EU restriction | restricted | does not fit anyway; license is a second reason to skip |
The takeaway is that every serveable pick in this catalog is Apache-2.0 or MIT, which is a deliberate consequence of the corrections: the picks that survived the fit test also happen to carry clean licenses, while the one license-encumbered family (Llama 4) is excluded on size first and license second.
The single most important correction is the Qwen one. Qwen3.6 is the open generation with downloadable weights under Apache-2.0; Qwen3.7 is API-only. The catalog serves local weights, so it tracks 3.6 and ignores 3.7 entirely until open weights appear. This is the same open-vs-closed distinction the early landscape doc missed when it listed "Qwen3.6/3.5" as if both were the local target.
The daily-driver slot wants interactive speed (well over 100 tok/s), strong general quality, and a comfortable fit in 22 GB. MoE wins this slot because A3B/A4B active counts mean decode reads only ~3-4B of weights per token while the full model stays resident. The speculative angle here is small: MoE is already near its bandwidth ceiling and a separate draft model usually hurts it (15 section 4), so the right play is a built-in MTP head where the family ships one, and nothing otherwise.
| Pick | Params (total/active) | Quant footprint in 22 GB | GGUF / EXL3 + HF source | Why | Speculative strategy |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B (default) | 35B / ~3B | UD-Q4_K_XL ~19-20 GB w/ 32K q8 KV | GGUF: unsloth/Qwen3.6-35B-A3B-GGUF, bartowski/Qwen_Qwen3.6-35B-A3B-GGUF. EXL3 lags. | Fastest strong general model that fits; ~120 tok/s derived at this quant. | Built-in MTP head (no separate draft model); easiest win, ~1.5-2x with high acceptance. |
| gpt-oss-20b | 21B / 3.6B | MXFP4 ~12-13 GB | GGUF: ggml-org / lmstudio-community gpt-oss-20b GGUF. | Smallest top-tier MoE; the ~9 GB of slack buys huge context headroom (128K+ in budget). | P-EAGLE / EAGLE3 head (Red Hat published one, see 7.5); MoE so no separate draft. |
| Gemma 4 26B-MoE | ~25.2B / ~3.8B | Q4_K_M ~17-18 GB | GGUF via Ollama gemma4:26b-a4b and bartowski; runs in Ollama cleanly. |
Multimodal daily driver with audio input; the Gemma family ships a matching MTP drafter. | Gemma 4 MTP 4-layer drafter (shared-KV with the target, ~3x vendor-reported; see 7.4). |
It is the largest model that still decodes at daily-driver speed on this box.
At UD-Q4_K_XL the weights are ~19-20 GB resident with a 32K q8 KV cache, which fits the 22 GB budget with a few hundred MB of margin (use UD-Q3_K_XL, ~16 GB, if you want long-context headroom; see section 10).
The active count is ~3B, so the roofline puts decode around 120 tok/s, in the same regime as the measured 217-240 tok/s for the older qwen3:30b-a3b anchor (05 section 1.2), discounted for the larger total and the dynamic quant.
The reason it does not need a separate draft model is in 15 section 4: speculation amortizes a large per-token weight read across several accepted tokens, but an A3B MoE already reads only ~3B of weights per token, so the read it would amortize is small and the extra draft forward passes plus the expert-routing churn usually net out flat or negative. Qwen3.6 ships a built-in multi-token-prediction (MTP) head, which predicts the next few tokens from the same forward pass with no second model loaded, so you get the amortization without paying for a draft. That is the strategy: turn on MTP, load no draft.
It is the cheapest way to keep a genuinely strong model resident with room to spare. MXFP4 native weights land at ~12-13 GB, which leaves ~9 GB for KV cache, so this is the pick when you want a long context window (agentic loops, big documents) without dropping to a smaller model. On Ada the MXFP4 blocks upcast to BF16 at load (section 11), so the footprint is the BF16-equivalent of the active path, but it still fits with enormous headroom. Speculation: gpt-oss has published EAGLE3-style heads (section 7.5), and because it is MoE you use the head, not a separate draft.
It is the daily driver you reach for when the conversation has images or audio in it. Ollama handles the Gemma 4 vision projector correctly (unlike Qwen3.6 vision, section 11), so this is the low-friction multimodal MoE. It is also the one daily-driver pick with a documented external MTP drafter (section 7.4): a 4-layer draft that shares the KV cache with the target and is reported at roughly 3x on Gemma 4 inference. That makes it the most interesting speculative case in this tier despite being slower than Qwen3.6-35B-A3B on raw decode.
Coding is the tier where speculative decoding matters most, because code is the most predictable token stream there is. Brackets, indentation, keyword sequences, and repeated identifiers give a draft model very high acceptance, which is why 15 section 3 quotes 1.8-2.9x on dense code. That is also why this tier splits by architecture: the fast MoE coder rides its own speed and an MTP head, while the dense coder is where a vocab-matched draft model pays off the most.
| Pick | Params (total/active) | Quant footprint in 22 GB | GGUF / EXL3 + HF source | Why | Speculative strategy |
|---|---|---|---|---|---|
| Qwen3-Coder-30B-A3B ("Coder-Flash") | 30B / ~3B | Q4_K_M ~18.6 GB; UD-Q5 if KV allows | GGUF: unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF, bartowski. | Best agentic local coder; ~70 tok/s measured; strong tool-call discipline. | MTP head or none (MoE; a separate draft hurts, see 15 section 4). |
| Qwen3.6-27B dense | 27B / 27B | Q4_K_M ~16.8 GB; Q6_K ~22 GB tight for code quality | GGUF via bartowski/unsloth; EXL3 via exllamav3. | Dense peak code quality; vendor-reported SWE-bench Verified 77.2. | Qwen3 1.7B draft (vocab-matched), 1.8-2.9x on code. |
| Devstral Small 2 24B | ~24B / ~6B | Q4_K_M ~15 GB | GGUF via Ollama devstral and bartowski. |
Apache-2.0-strict agentic coder; SWE-bench-tuned, small footprint. | MTP/none if served as MoE; otherwise vocab-matched Mistral draft. |
It is the model you run for agentic coding loops where throughput compounds across many short turns. Q4_K_M lands at ~18.6 GB resident (unsloth and bartowski both publish clean GGUFs), and the measured decode for the family is ~70-87 tok/s (05 section 1.2), slower than the general MoE because coding fine-tunes route experts less coherently but still well into interactive territory. Because it is an A3B MoE, the speculation rule from 15 section 4 applies: a separate draft model usually nets flat or negative, so you use the built-in MTP head if the runtime exposes it for this checkpoint and otherwise run plain. Do not bolt a Qwen3 0.6B/1.7B draft onto this MoE expecting the dense speedup; it does not transfer.
Unsloth's writeup of this model claims it "beats" a 397B-class model on some coding evals; that figure is vendor-reported and directional, not box-measured, and the catalog tracks it as "best local agentic coder that fits," not as a frontier-beater.
This is the dense coder, and dense is exactly the architecture speculative decoding was built for. At Q4_K_M the weights are ~16.8 GB, which leaves room to pair it with a small draft model and still fit a usable code context; if you want maximum code fidelity, Q6_K is ~22 GB and is tight but possible with a short context and q8 KV (section 10). Vendor-reported SWE-bench Verified is 77.2, the strongest local dense coding number in this catalog.
The draft pairing is the point. Qwen3.6-27B shares the Qwen3 tokenizer/vocabulary with the small Qwen3 dense models, so a Qwen3 1.7B draft is vocab-matched and legal (section 7.6 on the hard vocab rule). On code that pairing lands in the 1.8-2.9x band from 15 section 3, which turns a ~43 tok/s dense decode into something that reads back like a fast model. Use Qwen3 1.7B over 0.6B here: section 7 explains why acceptance beats raw draft speed.
It is the pick when license strictness or a smaller footprint matters more than peak SWE-bench score. At ~15 GB it leaves the most KV headroom of the three, which is useful for long agentic contexts. If the runtime treats it as a dense model, pair it with a vocab-matched Mistral-family draft; if it is served with its MoE routing active, use MTP/none per the same MoE rule.
Reasoning models spend most of their tokens on a long internal chain before the answer, so raw decode speed matters more here than anywhere else: the user waits through the whole /think trace.
That makes both fast MoE reasoning and speculative dense reasoning attractive, but with a caveat: reasoning traces are less predictable than code, so draft acceptance is lower and the speculative speedup is smaller than the coding numbers.
| Pick | Params (total/active) | Quant footprint in 22 GB | GGUF / EXL3 + HF source | Why | Speculative strategy |
|---|---|---|---|---|---|
| Qwen3.6-27B /think | 27B / 27B | Q4_K_M ~17 GB w/ q8 KV | GGUF via bartowski/unsloth; EXL3 via exllamav3. | Strongest local reasoning that fits; vendor-reported GPQA-Diamond 87.8. | Qwen3 1.7B draft (vocab-matched), modest speedup on reasoning traces. |
| gpt-oss-20b (high effort) | 21B / 3.6B | MXFP4 ~12-13 GB | GGUF: ggml-org / lmstudio-community. | Reasoning-tuned with selectable effort; fast MoE, big ctx for long traces. | EAGLE3 head (section 7.5); MoE so no separate draft. |
| Qwen3.6-35B-A3B (thinking) | 35B / ~3B | UD-Q4_K_XL ~19-20 GB | GGUF: unsloth, bartowski. | Daily driver in thinking mode; fastest long-trace decode in the tier. | Built-in MTP head; no separate draft. |
The speed gap across this tier is the reason it has three picks instead of one. A dense 27B reasoning model decodes in the ~43 tok/s band (05 section 1.2), so a 2000-token reasoning trace takes the better part of a minute before the answer appears; the same trace on the A3B MoE at ~120 tok/s lands in well under half that. Speculation narrows the dense gap on the predictable spans of a trace (boilerplate, restated problem text, arithmetic scaffolding) but cannot rescue the genuinely branchy reasoning steps where acceptance falls, so the dense pick stays the quality choice and the MoE picks stay the latency choices.
This is the reasoning pick when you want the best answer and will tolerate dense decode speed. Q4_K_M is ~17 GB with a q8 KV cache, fitting comfortably, and the vendor-reported GPQA-Diamond is 87.8, the top reasoning score among models that fit this box. Because it is dense and Qwen-vocab, it takes the same Qwen3 1.7B draft as the dense coder. The speedup is real but smaller than code: reasoning traces have more genuine branching, so acceptance is lower, and you should expect the lower end of the speculative band rather than the 2.9x ceiling.
This is the reasoning pick when latency matters more than the last point of accuracy. It exposes a reasoning-effort control, so you can dial the trace length, and at ~12-13 GB it leaves room for the long contexts that reasoning traces consume. As a MoE it uses its EAGLE3 head (section 7.5), not a separate draft. This is the practical default for interactive reasoning where you want an answer in seconds, not after a minute of dense decode.
If your daily driver is already Qwen3.6-35B-A3B, you do not need to swap models to get reasoning; switch it into thinking mode. This is the fastest long-trace decoder in the tier because the A3B active count keeps decode near daily-driver speed even through a long chain, and it rides the same built-in MTP head. The trade is quality: the dense 27B reasons better on hard benchmarks, but the MoE finishes the trace far sooner.
This tier is about answer quality per token, not speed. Dense models read every weight every token, so they are slow on this box (the ~25-42 tok/s band from 15), but at a given parameter count a dense model is usually sharper than a same-size MoE because all parameters participate. Every pick here is a candidate for the dense speculative treatment, because dense is the architecture where a vocab-matched draft turns ~43 tok/s into something usable.
| Pick | Params | Quant footprint in 22 GB | GGUF / EXL3 + HF source | Why | Speculative strategy |
|---|---|---|---|---|---|
| Qwen3.6-27B dense | 27B | Q4_K_M ~16.8-17 GB; Q6_K ~22 GB tight | GGUF via bartowski/unsloth; EXL3 via exllamav3. | Best all-round dense that fits; strong on code, reasoning, general. | Qwen3 1.7B draft (vocab-matched). |
| Mistral Small 4 24B | 24B | Q4_K_M ~16 GB | GGUF via Ollama mistral-small and bartowski. |
Apache-2.0 dense, multimodal, clean instruction following. | Vocab-matched Mistral-family draft (e.g. a small Mistral/Ministral). |
| Qwen3-32B dense | 32B | Q4_K_M ~19 GB; manage ctx | GGUF via bartowski; EXL3 4bpw via exllamav3. | Largest dense that fits; peak dense quality, watch the KV budget. | Qwen3 1.7B draft (vocab-matched). |
It is the one dense model in the catalog that is good at everything: it carries the coding tier's SWE-bench 77.2, the reasoning tier's GPQA 87.8, and strong general quality, all in a 27B dense body that fits at ~16.8 GB Q4_K_M. Q6_K (~22 GB) is available if you want maximum fidelity and will keep the context short with q8 KV, but Q4_K_M plus a Qwen3 1.7B draft is the better operating point because the draft buys back the speed the dense weights cost. This is the same model that appears in tiers 2, 3, and 5; it is the workhorse of this catalog.
It is the dense pick when you want Mistral's instruction style or a multimodal dense model under a clean Apache-2.0 license. At ~16 GB it fits with room for a vocab-matched Mistral-family draft. It is also the fallback for tier 5 vision when you want a dense (not MoE) multimodal model.
This is the ceiling of the dense tier: at Q4_K_M the weights are ~19 GB, so the KV cache budget is tight and you must keep context modest (16-32K) with q8 KV or it spills. It buys a few points of dense quality over the 27B at the cost of context room and speed. EXL3 4bpw via ExLlamaV3 is the better runtime here (15 section 6 reports ExLlamaV3 ~12% faster than GGUF Q4 at this size), and it pairs with the same Qwen3 1.7B draft. Reach for it only when you specifically need the extra dense capacity and can live with the context constraint.
Vision adds a wrinkle that has nothing to do with the model and everything to do with the runtime: the vision projector (the mmproj weights that turn image patches into tokens) has to be loaded and wired correctly, and not every runtime does it right.
The hard operational rule for this tier is in section 11: Ollama mishandles the Qwen3.6 vision mmproj, so Qwen vision goes through llama.cpp or LM Studio, while Gemma 4 vision is fine in Ollama.
| Pick | Params (total/active) | Quant footprint in 22 GB | GGUF / EXL3 + HF source | Why | Speculative strategy |
|---|---|---|---|---|---|
| Qwen3.6-27B (native vision) | 27B / 27B | Q4_K_M ~17 GB + mmproj | GGUF via bartowski/unsloth; use llama.cpp / LM Studio, NOT Ollama. | Native vision in the workhorse dense model; one model for text + images. | Qwen3 1.7B draft on the text stream (vocab-matched). |
| Qwen3-VL-32B | 32B / 32B | Q4_K_M ~19 GB + mmproj; manage ctx | GGUF + EXL3 via github.com/QwenLM/Qwen3-VL. | Dedicated VL model; strongest local vision quality. | Qwen3 1.7B draft (vocab-matched); watch KV budget. |
| Gemma 4 26B-MoE | ~25.2B / ~3.8B | Q4_K_M ~17-18 GB + mmproj | GGUF via Ollama gemma4:26b-a4b; runs in Ollama. |
Multimodal MoE with audio input; low-friction in Ollama. | Gemma 4 MTP 4-layer drafter (section 7.4). |
| Phi-4-reasoning-vision-15B | 15B / 15B | Q4_K_M ~10 GB + mmproj | GGUF via bartowski; HF microsoft/Phi-4-reasoning-vision-15B. | Small reasoning-vision model; big KV headroom, MIT. | Small Phi-family draft if vocab-matched; else none. |
It is the model that lets you run text and vision through one checkpoint without swapping.
At Q4_K_M plus the vision projector it is ~17 GB, fits cleanly, and inherits the dense model's strong reasoning, so it is a genuinely capable image-reasoning model rather than a captioner.
The catch is the runtime: serve it through llama.cpp or LM Studio, never Ollama, because Ollama mishandles the Qwen3.6 mmproj and silently degrades or drops the image path (section 11).
On the text stream it takes the same Qwen3 1.7B draft as the dense coder.
When the job is hard visual reasoning (charts, documents, dense diagrams), the dedicated VL model is the strongest local option. It ships both GGUF and EXL3 from the QwenLM/Qwen3-VL repo, so you have a runtime choice. At ~19 GB plus the projector the KV budget is tight, same caution as the dense 32B: keep context modest. Vocab-matched Qwen3 1.7B draft applies on the text path.
This is the one to reach for when you want multimodal without fighting the runtime. It runs in Ollama with the projector handled correctly, it adds audio input on top of vision, and as a ~3.8B-active MoE it decodes fast. It also carries the Gemma 4 MTP 4-layer drafter (section 7.4), making it the only multimodal pick with a documented speculative path.
At 15B it is the lightweight vision option: ~10 GB at Q4_K_M leaves enormous KV headroom for long multi-image contexts, and it carries Phi's reasoning fine-tune under a clean MIT license. This is the pick for vision tasks that do not need 27-32B capacity and benefit from the leftover VRAM.
This is the throughput tier and it carries more weight than its size suggests. These models do two jobs: they serve directly when you want a tiny fast model, and, more importantly, they are the draft models that make the dense picks in tiers 2-5 usable. The whole speculative-decoding decision from 15 and 16 lives or dies on getting this tier right, so it gets the most detail.
| Pick | Params | Footprint | GGUF / EXL3 + HF source | Role | Notes |
|---|---|---|---|---|---|
| Qwen3-1.7B | 1.7B | ~1.3 GB Q4 | GGUF via bartowski/unsloth; EXL3 via exllamav3. | Best Qwen-family draft model. | Beats 0.6B as a draft despite being slower (see 7.2). |
| Qwen3-0.6B | 0.6B | ~0.5 GB Q4 | GGUF via bartowski/unsloth. | Faster draft, lower acceptance. | ~60% lower acceptance than 1.7B; coexists on the card. |
| Gemma 4 MTP 4-layer drafter | ~4 layers | shares target KV | Gemma 4 release; see buildfastwithai writeup. | MTP drafter for Gemma 4 targets. | Shared-KV with target, ~3x vendor-reported. |
| Qwen3.6 built-in MTP head | n/a (in target) | n/a | Ships inside Qwen3.6 checkpoints. | MTP for Qwen3.6 targets. | No separate model to load; easiest win. |
| gpt-oss P-EAGLE / EAGLE3 head | EAGLE head | small | Red Hat published P-EAGLE for gpt-oss. | EAGLE3 speculation for gpt-oss. | MoE so no separate draft model. |
A small model can serve directly: Qwen3-1.7B and Qwen3-0.6B both run as standalone fast models for trivial tasks, classification, and the gateway's optional fast-path routing (05 section 12.3). But the higher-value job is drafting. A draft model proposes several tokens cheaply, the large target verifies them in one forward pass, and every accepted token was produced at the draft's speed instead of the target's (15 section 3). The arithmetic only works if acceptance is high, which is why the choice of draft is not "smallest and fastest."
This is the single most counterintuitive call in the tier, so it gets a derivation. Speculative throughput is governed by acceptance rate, not draft speed. The effective speedup is roughly the number of tokens accepted per verification step, capped by how cheap the draft is relative to the target.
Qwen3-0.6B drafts faster, but it is a weaker model, so the target rejects more of its proposals. Qwen3-1.7B drafts slower per token, but its proposals look much more like the target's own distribution, so acceptance is far higher. Field reports and the Qwen-family draft pairings put the 0.6B at roughly 60% lower acceptance than the 1.7B on general and code workloads (third-party-reported, draft-pairing writeups; treat as directional).
Work the trade. A draft that is rejected costs a wasted draft forward pass and contributes zero accepted tokens. Past a point, making the draft faster but less accurate lowers throughput because you spend more verification steps recovering from rejections. On this box, against a 27-32B dense target, the 1.7B draft sits on the right side of that trade: its extra draft cost is small next to a ~19 GB dense verify, and its higher acceptance means more tokens land per verify. So the rule is: use Qwen3-1.7B as the default Qwen draft, and only drop to 0.6B if you are VRAM-starved and need the ~0.8 GB back.
This is the table the rest of the catalog points at. Each large target is paired with the draft or MTP head it should speculate against, with the expected speedup band. Speedups are from 15 section 3 (box-derived for the dense band) and vendor-reported where noted.
| Target | Recommended draft / MTP | Vocab match | Expected speedup |
|---|---|---|---|
| Qwen3.6-35B-A3B (MoE) | Built-in MTP head | n/a (in-target) | ~1.5-2x (MTP), no separate draft |
| Qwen3-Coder-30B-A3B (MoE) | MTP head or none | n/a | ~1.3-1.8x on code (MTP); separate draft hurts |
| Qwen3.6-27B dense | Qwen3-1.7B | Qwen3 vocab (match) | 1.8-2.9x on code, lower on reasoning |
| Qwen3-32B dense | Qwen3-1.7B | Qwen3 vocab (match) | 1.8-2.9x on code |
| Qwen3-VL-32B (text path) | Qwen3-1.7B | Qwen3 vocab (match) | ~1.5-2.5x on text spans |
| Mistral Small 4 / Devstral 24B | small Mistral-family draft | Mistral vocab (must match) | ~1.5-2x on code |
| Gemma 4 26B-MoE / 31B | Gemma 4 MTP 4-layer drafter | shared with target | ~3x (vendor-reported) |
| gpt-oss-20b (MoE) | EAGLE3 / P-EAGLE head | n/a (in-target) | ~1.5-2x (vendor-reported) |
Gemma 4 ships a dedicated MTP drafter built as a small 4-layer head that shares the KV cache with the target model. Sharing the KV cache is the trick: the draft does not maintain its own context state, so it is cheap to run and stays in lockstep with the target's attention. The buildfastwithai writeup reports roughly 3x faster inference on Gemma 4 with this drafter enabled (third-party-reported: www.buildfastwithai.com/blogs/gemma-4-mtp-drafter-faster-inference; vendor/community-reported, directional). Because it is purpose-built for the Gemma 4 target it sidesteps the vocab problem entirely.
Red Hat published a P-EAGLE speculative head for gpt-oss that does EAGLE3-style speculation (developers.redhat.com/articles/2026/04/16, gpt-oss P-EAGLE). EAGLE-style heads predict from the target's own hidden states rather than running a separate full model, so like MTP they avoid loading a second model and avoid the vocab mismatch class of bug. For gpt-oss-20b this is the speculative path; there is no separate draft model to pair.
This rule is non-negotiable and it is where most speculative-decoding setups silently break. The draft and the target must use the same tokenizer and the same vocabulary. Speculation works by having the target verify the draft's proposed token IDs; if the two models map text to different token IDs, the IDs the draft proposes do not mean the same thing to the target, and the verification is comparing apples to a different vocabulary's oranges.
The failure mode is silent, which is what makes it dangerous. A cross-vocab pairing does not error out. It corrupts structured output: JSON keys drift, code tokens come out malformed, stop sequences miss, and the damage looks like a flaky model rather than a misconfiguration. A concrete example from this family: a mismatched Qwen3-0.6B paired against a non-Qwen target silently fails to build the draft context at all, so you get zero speedup and degraded output with no log line explaining why.
The operational consequence:
- Qwen3.6-27B / Qwen3-32B / Qwen3-VL-32B speculate only against Qwen3-family drafts (1.7B, 0.6B). Same Qwen3 vocab.
- Mistral / Devstral targets speculate only against Mistral-family drafts.
- Gemma 4 targets use the Gemma 4 MTP drafter, built for that vocab.
- gpt-oss uses its own EAGLE3 head.
- Never pair across families. A Qwen draft on a Mistral target, or vice versa, is the corruption case above.
The gateway config (section 12) carries the draft pairing as an explicit field so this rule is enforced in configuration, not left to whoever starts the runtime.
To make the 1.7B-over-0.6B call concrete, write down the throughput model speculation actually follows.
Let the target decode at T tok/s standalone, let the draft propose k tokens per verification step, and let a be the fraction of proposed tokens the target accepts.
Each verification step costs roughly one target forward pass plus k cheap draft passes.
The accepted tokens per verification step are approximately 1 + a*k (the one guaranteed token from the verify, plus the accepted run of the draft).
speculative_tok/s ≈ T × (1 + a*k) / (1 + k*draft_cost_ratio)
draft_cost_ratio is the draft's per-token cost as a fraction of the target's, which for a 1.7B draft against a ~19 GB dense target is small, on the order of a few percent.
The numerator is what acceptance buys you; the denominator is what the draft passes cost.
Because draft_cost_ratio is tiny for both 0.6B and 1.7B against a 19 GB target, the denominator barely moves between them, but the numerator moves a lot: pushing a from roughly 0.4 (0.6B) to roughly 0.65 (1.7B) at k=4 raises 1 + a*k from ~2.6 to ~3.6, a ~38% throughput swing in favor of the more accurate draft.
That is the whole reason the catalog defaults to 1.7B: against a heavy dense target the draft cost is in the noise, so you optimize acceptance, and acceptance is what the larger draft buys.
The story flips only when the target is light enough that draft_cost_ratio stops being negligible, which is exactly the MoE case in 7.7.
Repeating the 15 section 4 result because it governs half this table: a separate draft model hurts MoE. The MoE already reads only its active experts per token, so the per-token read speculation would amortize is small, and the draft's own forward passes plus the expert-routing disruption usually net flat or negative. Built-in MTP heads avoid this because they piggyback on the target's own forward pass with no second model and no extra routing. So every MoE target in this catalog (Qwen3.6-35B-A3B, Qwen3-Coder-30B-A3B, Gemma 4 26B-MoE, gpt-oss-20b) speculates with its head, never with a Qwen3-1.7B-style separate draft.
Embeddings are encode-only, so there is no decode tok/s and no speculative decoding here; the picks are about retrieval quality, multilingual coverage, footprint, and license. The Qwen3-Embedding family is the default because it tops the multilingual retrieval leaderboards and ships in three sizes that coexist cleanly on the card.
| Pick | Params | Footprint | Source | Why |
|---|---|---|---|---|
| Qwen3-Embedding-4B (default) | 4B | ~8 GB | ollama.com/library/qwen3-embedding; HF Qwen/Qwen3-Embedding-4B | Sweet spot of quality and footprint; strong multilingual retrieval. |
| Qwen3-Embedding-8B | 8B | ~16 GB | HF Qwen/Qwen3-Embedding-8B | Vendor-reported #1 on MTEB multilingual; peak retrieval quality. |
| Qwen3-Embedding-0.6B | 0.6B | ~1 GB | ollama.com/library/qwen3-embedding | Tiny, coexists with a large model on the card; high-throughput indexing. |
| BGE-M3 | ~0.6B | ~0.5 GB | HF BAAI/bge-m3 | MIT-licensed multilingual; the clean-license fallback. |
It is the best quality-per-GB embedding model that still leaves room for a serving model on the card. At ~8 GB it can coexist with a small generation model or run alongside a hot-swap target, and its multilingual retrieval quality is at or near the top of the open field. This is the embedding model the RAG paths in 13 should target by default.
When retrieval quality is the whole game and the box is dedicated to indexing or a retrieval service, the 8B is the vendor-reported #1 on MTEB multilingual. At ~16 GB it largely owns the card, so run it when embeddings are the primary workload, not alongside a large generator.
The 0.6B is the pick when you need embeddings to share the card with a large model, or when you are bulk-indexing and throughput matters more than the last point of recall. At ~1 GB it fits in the margin of almost any generation setup.
BGE-M3 is the fallback when the Apache-2.0 Qwen license is not acceptable and you want an MIT-licensed multilingual embedder. It is small, well-understood, and supports dense, sparse, and multi-vector retrieval in one model.
These are the 2026 frontier-class open models. They are documented here precisely because the early landscape doc named several of them as if they were candidates, and they are not. The 24 GB card cannot hold any of them; they spill to system RAM over PCIe and run 5-10x slower, which on a 400B-1T model means single-digit or sub-1 tok/s. They are the ceiling the box is measured against, not what it serves.
| Model | Params (total / active) | Why it does not fit | Source |
|---|---|---|---|
| DeepSeek-V4-Pro | 1.6T / 49B active | 1.6T weights need hundreds of GB even at 4-bit; ~49B active alone exceeds the card. | api-docs.deepseek.com/news/news260424 |
| DeepSeek-V4-Flash | 284B / 13B active | 284B total needs ~140 GB+ at 4-bit; total drives VRAM, not the 13B active. | api-docs.deepseek.com/news/news260424 |
| GLM-5.2 | 744B / 40B active | 744B total, vendor-reported #1 open on Artificial Analysis; no vision; ~370 GB+ at 4-bit. | latent.space (GLM-5.2, vendor-reported) |
| Kimi K2.6 (general flagship) | ~1T / 32B active | ~340 GB+ resident; impossible on 24 GB. | nerova.ai (Kimi coverage) |
| Kimi K2.7-Code (coding) | 1T / 32B active | Coding-specialized 1T; ~340 GB+; impossible. Note this is the June K2.7 release, not a general model. | nerova.ai (Kimi K2.7-Code) |
| Mistral Large 3 | 675B | 675B dense-class; hundreds of GB; far over budget. | mistral.ai/news/mistral-3 |
| Llama 4 Scout | 109B (smallest L4) | Even the smallest Llama 4 is 109B; ~55 GB+ at 4-bit; EU-restricted license on top. | huggingface.co/blog/llama4-release |
| Llama 4 Maverick | larger than Scout | Larger than Scout; dated; EU-restricted. | huggingface.co/blog/llama4-release |
| gpt-oss-120b | 120B / ~5B active | 120B total needs ~80 GB even at MXFP4; the 20b is the one that fits, not this. | openai.com/index/introducing-gpt-oss |
Every entry here is a MoE except Mistral Large 3 and Llama, and the same mistake recurs: people see "49B active" or "13B active" or "32B active" and assume the active count is the VRAM number. It is not. The full set of experts has to be resident in VRAM; the active count only governs decode speed (05 section 1.4). DeepSeek-V4-Flash at 13B active still needs all 284B parameters in memory. This is why "small active count" frontier MoEs are still impossible on a 24 GB card.
Naming the ceiling keeps the catalog honest. When a user asks "can the box run GLM-5.2" or "what about DeepSeek-V4," the answer is a documented no with the arithmetic, not a vague "it is too big." It also frames the local tier correctly: the box serves the ~30B class, and the ~30B class is a deliberate floor under the 400B-1T frontier, chosen because it is the best quality that fits the bandwidth and VRAM of one 4090 (01).
The quant choice is where the 22 GB budget gets spent, and 15 section 5 made the case for Unsloth dynamic GGUFs as the best quality-per-byte on this box. This section turns that into per-tier guidance.
Unsloth's dynamic quants (the UD- series) vary bits per tensor: they keep sensitive tensors (attention, the first and last blocks, routing) at higher precision and push the bulk of the weights lower. The result is meaningfully better quality at a given footprint than a flat K-quant (15 section 5, 04). For the 30-35B MoE daily driver, the two operating points are:
- UD-Q3_K_XL ~16 GB: full residency with comfortable KV headroom on a 30-35B MoE. Use this when you want long context or a coexisting embedding model.
- UD-Q4_K_XL ~19-21 GB: tighter but higher quality. Use this when the model is the only large thing on the card and you keep context at 32K with q8 KV.
Coding quality degrades faster under aggressive quantization than general chat, because a single wrong token in code is a syntax error, not a slightly-off word. So for the dense coders (Qwen3.6-27B, Qwen3-32B) prefer Q5_K_M or Q6_K when it fits, and only drop to Q4_K_M when the footprint forces it. The trade is direct: Q6_K Qwen3.6-27B is ~22 GB and tight, but if you can keep context short with q8 KV it is the highest-fidelity local coder configuration. For the MoE coder (Qwen3-Coder-30B-A3B) Q4_K_M is the practical default because the MoE footprint is already near the budget; step up to UD-Q5 only if the KV math allows.
The 22 GB line is a cliff (05 section 1.4): one spilled layer drops you 5-10x. When a model's full context window would push KV past the budget, cap context at 32-64K or quantize the KV cache to q8 (or both). q8 KV roughly halves the KV footprint at near-zero quality cost (15, 04), which is almost always the right first move before you drop the weight quant. Order of operations when a config does not fit: q8 KV first, then cap context, then drop weight quant, in that order, because each step costs less quality than the next.
| Model | Quant | Footprint (weights + 32K q8 KV) | Fits 22 GB |
|---|---|---|---|
| Qwen3.6-35B-A3B | UD-Q3_K_XL | ~16-17 GB | yes, with headroom |
| Qwen3.6-35B-A3B | UD-Q4_K_XL | ~19-21 GB | yes, tight |
| gpt-oss-20b | MXFP4 (BF16 upcast on Ada) | ~12-14 GB | yes, large headroom |
| Qwen3-Coder-30B-A3B | Q4_K_M | ~18.6 GB | yes |
| Qwen3.6-27B dense | Q4_K_M | ~16.8-17 GB | yes |
| Qwen3.6-27B dense | Q6_K | ~22 GB | tight, short ctx only |
| Qwen3-32B dense | Q4_K_M | ~19 GB | yes, manage ctx |
| Qwen3-Embedding-4B | Q8/FP16 | ~8 GB | yes (coexists) |
These are the traps that have actually bitten on this box or are documented well enough to plan around. They are operational, not theoretical, and each one has a concrete workaround.
Qwen3.6 low-bit GGUFs (Q3/Q4 class) produce garbled output on CUDA 13.2 specifically. The fix is to pin the CUDA toolkit to 13.1 or 13.3, or run a newer llama.cpp build that carries the fix for the 13.2 kernel regression. This is a toolchain bug, not a model or quant bug: the same GGUF is fine on the adjacent CUDA versions. Verify the CUDA version the runtime actually links against (not just what is installed) before blaming the quant.
Ollama does not wire the Qwen3.6 vision projector correctly, so the image path silently degrades or drops.
Serve Qwen3.6 vision through llama.cpp or LM Studio, which handle the mmproj correctly.
Gemma 4 vision is fine in Ollama; this is specifically a Qwen3.6-vision-in-Ollama problem.
This is why tier 5 splits the runtime recommendation by model.
The box's CDN reproducibly stalled the qwen3-coder download, documented in 09.
Plan for it: pull qwen3-coder during a maintenance window, expect to retry, and consider pulling the GGUF directly from HuggingFace (unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) as the fallback path rather than relying on the stalling CDN route.
See 09 for the full incident and the retry runbook.
gpt-oss ships in MXFP4, but native MXFP4 hardware support is Hopper and Blackwell. The 4090 is Ada (sm_89), so the MXFP4 blocks upcast to BF16 at load. gpt-oss still runs fine on Ada; the only consequence is that the footprint is the BF16-equivalent of the active path rather than the true 4-bit footprint, which is already accounted for in the ~12-14 GB figure in section 10.4. Do not expect the tiny on-disk MXFP4 size to be the VRAM size on this box.
Before a model joins the default set, run this five-point check so the gotchas above do not surface in production:
- Confirm the CUDA version the runtime links against is 13.1 or 13.3 (not 13.2) for any Qwen3.6 low-bit quant (11.1).
- For vision picks, send one image-bearing request and confirm the projector path returns a grounded answer, not a hallucinated one; do this on the exact runtime the gateway will use (11.2).
- Confirm the pull completed and the blob SHA256 matches the manifest entry; for qwen3-coder, expect to use the HuggingFace fallback if the CDN stalls (11.3).
- For gpt-oss, confirm the resident VRAM matches the ~12-14 GB BF16-upcast figure, not the smaller on-disk MXFP4 size (11.4).
- For any dense pick with a draft, confirm the draft's
vocabfield matches the target family before enabling speculation; a silent corruption here is the worst failure mode in the catalog (7.6).
Record the result of each check in the manifest notes field so the next person does not re-discover the same trap.
The default set the box installs and serves, refreshed to 2026-06-23 names and paired with their speculative strategy.
| Slot | Pick | Quant | Speculative strategy | Gateway id |
|---|---|---|---|---|
| Daily driver | Qwen3.6-35B-A3B | UD-Q4_K_XL | built-in MTP | daily-driver |
| Coder (fast) | Qwen3-Coder-30B-A3B | Q4_K_M | MTP / none | coder |
| Coder (peak) | Qwen3.6-27B dense | Q5_K_M/Q6_K | Qwen3-1.7B draft | coder-dense |
| Reasoning | Qwen3.6-27B /think | Q4_K_M | Qwen3-1.7B draft | reason |
| Peak dense | Qwen3.6-27B dense | Q4_K_M | Qwen3-1.7B draft | dense |
| Vision | Qwen3.6-27B vision (llama.cpp) | Q4_K_M | Qwen3-1.7B draft (text) | vision |
| Vision (low-friction) | Gemma 4 26B-MoE | Q4_K_M | Gemma 4 MTP drafter | vision-moe |
| Small/fast | gpt-oss-20b | MXFP4 | EAGLE3 head | fast |
| Draft | Qwen3-1.7B | Q4_K_M | (is the draft) | draft |
| Embeddings | Qwen3-Embedding-4B | Q8 | n/a (encode) | embed |
The picks above map to the gateway's manifest and routing config.
The model manifest (catalog/models.manifest.yaml, schema in 10 section 11) declares what is installed; the routing config (configs/llmgw.yaml, schema in 08) declares the ids clients send.
The new field this doc adds is draft: each large entry names its draft model id or mtp/eagle3 for an in-target head, and the gateway enforces the vocab-match rule (section 7.6) by refusing to wire a cross-family draft.
# configs/llmgw.yaml (excerpt; full schema in 08)
models:
- name: daily-driver
alias_for: qwen3.6:35b-a3b
runtime: inproc # in-process llama.cpp, doc 16
quant: UD-Q4_K_XL
context: 32768
kv_cache_type: q8_0
speculative: { mode: mtp } # built-in MTP head, no draft model
- name: coder-dense
alias_for: qwen3.6:27b
runtime: inproc
quant: Q5_K_M
context: 32768
kv_cache_type: q8_0
speculative: { mode: draft, draft: qwen3-1.7b } # vocab-matched
- name: coder
alias_for: qwen3-coder:30b-a3b
runtime: inproc
quant: Q4_K_M
context: 65536
kv_cache_type: q8_0
speculative: { mode: none } # MoE; separate draft hurts (15 section 4)
- name: vision
alias_for: qwen3.6:27b-vision
runtime: llamacpp # NOT ollama: mmproj gotcha (11.2)
quant: Q4_K_M
context: 32768
kv_cache_type: q8_0
speculative: { mode: draft, draft: qwen3-1.7b }
- name: fast
alias_for: gpt-oss-20b
runtime: inproc
quant: MXFP4
context: 131072
speculative: { mode: eagle3 }
- name: embed
alias_for: qwen3-embedding-4b
runtime: ollama
endpoint: /v1/embeddings
draft_models:
- id: qwen3-1.7b
vocab: qwen3 # gateway refuses to pair across vocab families
source: bartowski/Qwen3-1.7B-GGUF
quant: Q4_K_MThe qwen3-1.7b draft is not a client-facing model; it appears under draft_models and is referenced by id from the targets that share its vocab, exactly as 05 section 12.1 said the draft is a runtime parameter, not a routing entry.
The manifest (catalog/models.manifest.yaml) records the exact source per pick so a fresh box reproduces the same set.
Prefer the Unsloth GGUFs for the dynamic-quant quality (section 10.1) and fall back to bartowski for breadth of quant levels.
| Gateway id | Source (HF repo or library tag) | Format / quant | Disk size (approx) |
|---|---|---|---|
daily-driver |
unsloth/Qwen3.6-35B-A3B-GGUF (UD-Q4_K_XL) | GGUF | ~20 GB |
coder |
unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF (Q4_K_M) | GGUF | ~18 GB |
coder-dense / dense / reason |
bartowski/Qwen_Qwen3.6-35B-A3B-GGUF and Qwen3.6-27B GGUF | GGUF | ~17-22 GB by quant |
vision |
Qwen3.6-27B vision GGUF + mmproj (bartowski/unsloth) | GGUF | ~17 GB + projector |
vision-moe |
Ollama gemma4:26b-a4b |
GGUF | ~17 GB |
fast |
ggml-org / lmstudio-community gpt-oss-20b GGUF | MXFP4 | ~12 GB |
draft |
bartowski/Qwen3-1.7B-GGUF (Q4_K_M) | GGUF | ~1.3 GB |
embed |
Ollama qwen3-embedding / HF Qwen/Qwen3-Embedding-4B |
GGUF | ~4-8 GB |
Verify the blob SHA256 after each pull and record it in the manifest, per 05 section 13.3 and 10 section 11; the family is the design decision, the point-release behind the tag is an operations concern (section 1, freshness caveat).
The default set never asks the card to hold two large models at once: the gateway hot-swaps the large slot (05 section 12.2) and only the small draft (~1.3 GB) and optionally embed (~8 GB) coexist with a target.
A worst-case co-resident check: daily-driver is MoE and uses MTP, so it loads no draft and sits at ~19-20 GB alone, leaving margin.
coder-dense at Q5_K_M (~19 GB) plus the draft (~1.3 GB) is ~20.3 GB plus KV, which is why that slot runs a shorter context with q8 KV (section 10.3).
Every co-resident pair was checked against the 22 GB line; the dense-plus-draft pairs are the tight ones and the MoE-plus-MTP pairs are the comfortable ones.
The decisions this refresh commits, each with its reason and the section that backs it.
DR-17.1: Qwen3.6 is the catalog spine, not Qwen3.7. Qwen3.6 (27B dense + 35B-A3B MoE) is the current open Apache-2.0 generation; Qwen3.7 is API-only with no downloadable weights. The box serves local weights, so the catalog tracks 3.6 and ignores 3.7 until open weights appear (section 1.3).
DR-17.2: The default daily driver is Qwen3.6-35B-A3B at UD-Q4_K_XL with its built-in MTP head. It is the largest model that still decodes at daily-driver speed, and MTP gives the speculative win with no second model and no vocab risk (sections 2.1, 7.7).
DR-17.3: MoE targets speculate with in-target heads, never a separate draft. A separate draft hurts MoE (15 section 4); Qwen3.6-35B-A3B, Qwen3-Coder-30B-A3B, Gemma 4 26B-MoE, and gpt-oss-20b all use MTP/EAGLE3 heads (section 7.7).
DR-17.4: Dense targets speculate with a vocab-matched draft; the default Qwen draft is Qwen3-1.7B, not 0.6B. Acceptance beats raw draft speed; 0.6B drafts ~60% lower acceptance, so 1.7B nets higher throughput against a ~19 GB dense verify (section 7.2).
DR-17.5: Draft and target must share a vocabulary, enforced in config.
Cross-vocab pairings silently corrupt structured output; the gateway refuses to wire a draft whose vocab does not match the target's family (sections 7.6, 12.1).
DR-17.6: Qwen3.6 vision is served via llama.cpp/LM Studio, never Ollama.
Ollama mishandles the Qwen3.6 mmproj (section 11.2); the routing config pins runtime: llamacpp for the vision id.
DR-17.7: gpt-oss-20b is in; gpt-oss-120b is out. The 20b (MXFP4, ~12-14 GB on Ada after BF16 upcast) fits with large headroom; the 120b needs ~80 GB and lives in the ceiling section (sections 2.2, 9, 11.4).
DR-17.8: The frontier models are the ceiling, not candidates. DeepSeek-V4-Pro/Flash, GLM-5.2, Kimi K2.6/K2.7-Code, Mistral Large 3, Llama 4 Scout/Maverick, and gpt-oss-120b do not fit 24 GB and are documented as the reference ceiling, with the total-vs-active arithmetic spelled out (section 9).
DR-17.9: Quant order of operations is q8 KV, then cap context, then drop weight quant. Each step costs less quality than the next; code stays at Q5_K_M/Q6_K when it fits because code is quant-sensitive (section 10).
DR-17.10: Vendor benchmark figures are directional. Qwen3.6 "beats 397B" on coding, GLM-5.2's #1-open index, the Gemma 4 ~3x MTP figure, and the MTEB rankings are vendor- or community-reported and are tracked as directional, not as box-measured truth (sections 1, 3.1, 7.4, 9).
- 01: the landscape survey this doc corrects, and the rationale for the ~30B local tier.
- 03: the VRAM and KV-cache arithmetic behind every footprint here.
- 04: the quant quality table and the per-tensor breakdown for the dynamic GGUFs in section 10.
- 05: the canonical catalog; column semantics, master table, gateway wiring, refresh cadence. This doc is its dated overlay.
- 08: the routing config schema (
configs/llmgw.yaml) that section 12 extends with thespeculativeanddraft_modelsfields. - 09: the qwen3-coder CDN stall incident (gotcha 11.3) and the measured anchors.
- 10: the model manifest schema (
catalog/models.manifest.yaml) and the pull runbook. - 13: the RAG paths that consume the tier 7 embedding picks.
- 15: the roofline, the speculative-decoding theory, the MoE-hurts result, and the dynamic-quant argument this doc applies.
- 16: the in-process llama.cpp engine (
runtime: inproc) that serves the speculative configs in section 12.
This doc is the dated overlay on 05. When the names move again, refresh this file, not 05; the column semantics, the budget contract, and the gateway wiring in 05 stay stable while the picks here roll with the releases.