162 billion parameters (8 bit quant) on one 40 GB card and tested on 2xH100 (2x80gb cards)
Last active
August 4, 2026 19:36
-
-
Save bigsnarfdude/2b05be835f983fc46c643ea4d80ea634 to your computer and use it in GitHub Desktop.
Getting Unsloth DeepSeek-V4-Flash-0731 GGUF running losslessly on a single rented A100 and 2xH100
Author
Author
The offload cliff
DeepSeek-V4-Flash-0731 measured across three tiers of hardware — and what a 284B model actually knows about upgrading our Rails app.
4 August 2026 · 2× H100 80GB SXM · single A10 23GB · llama.cpp 071327508 · unsloth UD-Q8_K_XL, 161,869,615,520 bytes
Three findings, in order of how much they'd change your buying decision:
1. Offloading 7% of expert layers to CPU costs 75% of prefill speed. Generation barely notices. They are two different bottlenecks on one knob.
2. A second GPU in llama.cpp buys capacity, not speed. At matched offload, two H100s perform identically to one — very slightly worse.
3. On a real upgrade task with a known answer key, it scores 4.75 / 7 — and the two it reliably misses are exactly the two that break the boot.
1 · Prefill falls off a cliff. Generation slides.
Everything below is the same model file on the same box. Only -ncmoe changes — how many of the 43 expert layers live on CPU instead of GPU.
-r 2, 2× H100 except the grey A10 rows. Bars normalized to each metric's own maximum.This is why the common advice — buy memory channels, buy RAM bandwidth — only ever fixed half the problem. It's correct for generation and irrelevant for prefill. With 41 of 43 layers on CPU, the CPU is doing the prefill matmuls, and no amount of DDR5 changes that.
The practical version
Coding agents open with a large system-and-tools preamble. Reasonix's is 13,215 tokens. Codex's is comparable.
A10 (ncmoe 41) 13,215 tok ÷ 46.6 tok/s = 284 seconds before the first byte 2×H100 (ncmoe 0) 13,215 tok ÷ 1445 tok/s = 9 seconds
At 284 seconds Codex gives up mid-request — the server log reads srv stop: cancel task. That's not a misconfiguration. It's the cliff.
2 · The second GPU is running a relay race
Two H100s at ~50% utilization each, drawing 290 W of a 700 W cap, looked wrong. llama.cpp's default --split-mode layer is a pipeline split: GPU0 runs its layers, hands off, GPU1 runs the rest. On single-stream decode they take turns.
The clean test is to hold offload constant and vary only card count:
config | prefill | generation -- | -- | -- 1× H100, -ncmoe 25 | 83.67 | 33.44 2× H100, -ncmoe 25 | 83.70 | 32.47 2× H100, -ncmoe 0 | 1445.94 | 56.97Power never exceeded 362 W of a 700 W cap per card, idling at 117 W. Generation waits on HBM, not on math — the 700 W envelope is there for prefill.
Agentic work leaves the GPU 40–50% idle regardless; it's latency-bound and bursty. Batch is the shape this hardware wants: eight concurrent slots took aggregate throughput from 57 to 139.6 tok/s. Which also describes the honest use case — not an assistant you sit in front of, but an oracle-scored batch job you read in the morning.
Corrections
Predictions I put on record before measuring, all wrong in the model's favour:
- Predicted 4/7 on the Gemfile eval, and specifically that
deep_cloneablewould be missed — it hits 95% with correct reasoning. - Called 2× H100 impractical because 162 GB "won't fit in 160 GB" — that was a GB/GiB error. The model is 150.75 GiB and fits with 8.5 GiB spare.
- Estimated prefill at 300–800 tok/s. Measured 1,445.
- Blamed Reasonix for the A10's zero cache hits. The H100 run shows the same retry event and still cached 40,200 tokens. unexplained
Open
- untested
--no-mmap— the loader explicitly warns that CPU tensor overrides plus mmap leaves performance on the table. - untested Speculative decoding via the MTP head. The 2× DGX Spark recipe measures 60.2% draft acceptance and 4.01 tokens/step — worth ~1.7× decode. Needs vLLM.
- unexplained Why the A10 cached nothing. Two variables differed; not isolated.
- the good one Does tool access rescue the knowledge gap? The 20-run eval was single-shot with no tools — it couldn't check whether liquid 3 runs on Ruby 3.2. Given
bundleand the repo it might discover the breakage empirically, which would flip "doesn't know" into "doesn't know but can find out." Completely different conclusion about usability.
Raw data, logs and all 20 eval outputs: ~/Downloads/deepseek/results-h100-2026-08-04/ and results-a10-2026-08-04/. Scoring was regex against the documented answer key and grep against a known-correct fix — no model judged another model's output.
Author
┌───────────────────────────────────────────────────────────┬──────────────────────────────────────┐
│ Decision │ Difficulty │
├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
│ rails 6.1.7.10 → 7.0.10 → 7.1.6, stepwise │ easy │
├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
│ ruby 2.7.8 → 3.2.8 │ easy │
├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
│ puma ~> 6.0 — Puma 5 refuses Rack 3 │ moderate, well-known │
├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
│ concurrent-ruby unpinned — pin only needed below 7.1 │ moderate (the Gemfile comment hints) │
├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
│ deep_cloneable ~> 3.2 — 3.1 caps ActiveRecord < 7 │ hard, gemspec trivia │
├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
│ liquid 3 → 4 — liquid 3 taint-checks, dead on Ruby 3.2 │ hard, obscure │
├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
│ tinymce-rails 5.10.2 → 5.10.9 — 5.10.2 calls File.exists? │ very hard, deeptrivia │
└───────────────────────────────────────────────────────────┴──────────────────────────────────────┘
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
162 billion parameters (8 bit quant) on one 40 GB card
DeepSeek-V4-Flash-0731 running losslessly on a single rented A100 — what it actually costs, and what it would take to host at home.
31 July 2026 · Lambda A100-SXM4-40GB · llama.cpp b1-876a432
How to read the numbers. Everything is tagged. measured was run on this box today. estimated is projected from bandwidth arithmetic and could be wrong by a factor of two. unverified is pricing or spec I haven't checked against current listings.
TL;DR
The setup
DeepSeek-V4-Flash is a mixture-of-experts model: 284B parameters total, but each token routes through 6 of its 256 experts plus one shared expert — about 13B parameters of actual work.
Think of it as a reference library. The building holds a quarter-million books, but answering any single question means pulling six off the shelf. You don't need the whole library on the desk — you need the desk to be fast, and the stacks merely need to be reachable.
That maps onto hardware almost exactly. The "desk" — attention, the shared expert, the KV cache — fits in 40 GB of VRAM. The "stacks" — 96% of the weights — sit in system RAM, and the CPU fetches six per token. The question was whether that fetch is fast enough to be usable.
What we did
1 · Verify the repo arch, real file sizesUnder an hour of wall clock, most of it download and CUDA build running in parallel. The key move is step 5 — splitting by tensor type rather than by layer:
-ngl 99sends every layer to the GPU;-ncmoe 37pulls the expert tensors of the first 37 layers back to CPU RAM. The blunt layer-level dial would drag all 256 experts along with every layer it moved — the tensor-level split is what makes this work at all.What we found
confirmed needs validation hit a wall open caveat
Expert placement: +12% prompt, +8% generation measured
Each expert layer is ~3.6 GB, so the cliff at 34 is exactly where you'd predict.
-ncmoe 37survives at 32k context with 4.5 GB of VRAM headroom.Throughput barely moves with context measured
This is the most useful result here. Generation is bottlenecked on streaming expert weights out of CPU RAM — a constant cost per token regardless of context length. Attention runs on the GPU and is a small enough share that growing the KV cache barely registers, helped by this model's MLA plus sparse attention (index_topk 512, sliding_window 128), which doesn't scale linearly with context.
A flat 15–17 is far more usable for agentic work than a 25 that decays to 6.
Two traps. Shard 00001-of-00005 is only 5.25 MB — a split header, not weights. Downloading just that file, which the obvious single-file URL invites, gets you nothing loadable.
And the first prompt-processing measurement was meaningless: llama.cpp's prompt cache had already absorbed the test prompt, so it reported 4 tokens instead of 1,800. All throughput figures here come from llama-bench, which sidesteps that.
Hosting this at home
The result generalizes into one uncomfortable conclusion: the thing worth buying is memory bandwidth and capacity, not a bigger GPU. Our card held 35 GB of a ~190 GB working set. Extrapolating the -ncmoe sweep, going 40 GB → 48 GB of VRAM buys about two more expert layers, or ~3%. That is a poor use of $4,000.
Where the 17 tok/s actually comes from
We're at roughly a quarter of peak bandwidth. That gap is the interesting part: it suggests the bottleneck isn't raw DRAM speed but the hybrid split itself — 43 layers of CPU↔GPU round trips, every single token. A unified-memory machine has no split and pays none of that overhead, which is a structural advantage on top of whatever bandwidth it has.
Platform options
2-channel DDR5
8-channel DDR4 — same class as this box
8-channel DDR5
unified memory, no split
"Max out the RAM" on a consumer board is the trap. You get capacity without bandwidth — a machine that loads the model and then generates at a third of what we measured. Worse, most consumer platforms cap at 192 GB (4×48 GB), which sits under unsloth's 169 GB recommendation once the OS takes its cut, and populating all four DDR5 slots usually forces the memory clock down — costing bandwidth exactly when you added capacity.
The 192 GB question
A used M2 Ultra Studio at ~$4k is the most interesting option on that table, and its constraint is knowable now without benchmarking anything — it's arithmetic.
192 GB total, minus ~12–16 GB for macOS, minus KV cache, against a 162 GB model. It fits. It does not fit comfortably, and you won't run large contexts. You'd also need to raise Metal's wired-memory limit — the default caps GPU allocation near 75% (~144 GB), which won't even load it:
So the real question isn't "192 GB or not" — it's which quant you're buying for:
Rent or buy
At roughly $1–2/hr for this class of instance, $4k is something like 3,000 hours — about 2.5 years at 5 hrs/day. Renting wins on intermittent use and skips the platform question entirely. Buying wins on always-on availability, silence, no metered clock, and data never leaving the house.
The argument against the M2 Ultra specifically is age: it's a 2023 part, and whether $4k used is fair in mid-2026 depends on the current lineup. If a newer Ultra with more unified memory is close in price, the 192 GB ceiling is the thing you'd regret — this model already needs 162 GB, and the next one will not be smaller.
What's next
Sources
Measured 31 July 2026. Throughput from llama-bench -r 2; depth series with -d 0,4096,16384,32768. Bandwidth figures for platforms other than this host are vendor peak specs, not measurements — treat the derived tok/s as order-of-magnitude only.