Getting Unsloth DeepSeek-V4-Flash-0731 GGUF running losslessly on a single rented A100 and 2xH100

gistfile1.md

162 billion parameters (8 bit quant) on one 40 GB card and tested on 2xH100 (2x80gb cards)

The offload cliff

DeepSeek-V4-Flash-0731 measured across three tiers of hardware — and what a 284B model actually knows about upgrading our Rails app.

4 August 2026 · 2× H100 80GB SXM · single A10 23GB · llama.cpp 071327508 · unsloth UD-Q8_K_XL, 161,869,615,520 bytes

Three findings, in order of how much they'd change your buying decision:

1. Offloading 7% of expert layers to CPU costs 75% of prefill speed. Generation barely notices. They are two different bottlenecks on one knob.

2. A second GPU in llama.cpp buys capacity, not speed. At matched offload, two H100s perform identically to one — very slightly worse.

3. On a real upgrade task with a known answer key, it scores 4.75 / 7 — and the two it reliably misses are exactly the two that break the boot.

1 · Prefill falls off a cliff. Generation slides.

Everything below is the same model file on the same box. Only -ncmoe changes — how many of the 43 expert layers live on CPU instead of GPU.

PREFILL (pp512, tok/s)GENERATION (tg128, tok/s)ncmoe 014453357623291711213641 (A10)4705735164794412414113Same knob. Two completely different curves.Generation waits on memory bandwidth — it degrades roughly linearly.Prefill waits on compute — the moment the CPU does expert matmuls, it collapses.Three of 43 layers offloaded (7%) → prefill drops 75%. Generation drops 11%.measured llama-bench, -r 2, 2× H100 except the grey A10 rows. Bars normalized to each metric's own maximum.

This is why the common advice — buy memory channels, buy RAM bandwidth — only ever fixed half the problem. It's correct for generation and irrelevant for prefill. With 41 of 43 layers on CPU, the CPU is doing the prefill matmuls, and no amount of DDR5 changes that.

The practical version

Coding agents open with a large system-and-tools preamble. Reasonix's is 13,215 tokens. Codex's is comparable.

A10   (ncmoe 41)   13,215 tok ÷  46.6 tok/s  =  284 seconds before the first byte
2×H100 (ncmoe 0)   13,215 tok ÷ 1445  tok/s  =    9 seconds

At 284 seconds Codex gives up mid-request — the server log reads srv stop: cancel task. That's not a misconfiguration. It's the cliff.

2 · The second GPU is running a relay race

Two H100s at ~50% utilization each, drawing 290 W of a 700 W cap, looked wrong. llama.cpp's default --split-mode layer is a pipeline split: GPU0 runs its layers, hands off, GPU1 runs the rest. On single-stream decode they take turns.

The clean test is to hold offload constant and vary only card count:

config | prefill | generation -- | -- | -- 1× H100, -ncmoe 25 | 83.67 | 33.44 2× H100, -ncmoe 25 | 83.70 | 32.47 2× H100, -ncmoe 0 | 1445.94 | 56.97

Power never exceeded 362 W of a 700 W cap per card, idling at 117 W. Generation waits on HBM, not on math — the 700 W envelope is there for prefill.

Agentic work leaves the GPU 40–50% idle regardless; it's latency-bound and bursty. Batch is the shape this hardware wants: eight concurrent slots took aggregate throughput from 57 to 139.6 tok/s. Which also describes the honest use case — not an assistant you sit in front of, but an oracle-scored batch job you read in the morning.

Corrections

Predictions I put on record before measuring, all wrong in the model's favour:

• Predicted 4/7 on the Gemfile eval, and specifically that deep_cloneable would be missed — it hits 95% with correct reasoning.
• Called 2× H100 impractical because 162 GB "won't fit in 160 GB" — that was a GB/GiB error. The model is 150.75 GiB and fits with 8.5 GiB spare.
• Estimated prefill at 300–800 tok/s. Measured 1,445.
• Blamed Reasonix for the A10's zero cache hits. The H100 run shows the same retry event and still cached 40,200 tokens. unexplained

Open

• untested --no-mmap — the loader explicitly warns that CPU tensor overrides plus mmap leaves performance on the table.
• untested Speculative decoding via the MTP head. The 2× DGX Spark recipe measures 60.2% draft acceptance and 4.01 tokens/step — worth ~1.7× decode. Needs vLLM.
• unexplained Why the A10 cached nothing. Two variables differed; not isolated.
• the good one Does tool access rescue the knowledge gap? The 20-run eval was single-shot with no tools — it couldn't check whether liquid 3 runs on Ruby 3.2. Given bundle and the repo it might discover the breakage empirically, which would flip "doesn't know" into "doesn't know but can find out." Completely different conclusion about usability.

Raw data, logs and all 20 eval outputs: ~/Downloads/deepseek/results-h100-2026-08-04/ and results-a10-2026-08-04/. Scoring was regex against the documented answer key and grep against a known-correct fix — no model judged another model's output.

  ┌───────────────────────────────────────────────────────────┬──────────────────────────────────────┐
  │                         Decision                          │              Difficulty              │
  ├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
  │ rails 6.1.7.10 → 7.0.10 → 7.1.6, stepwise                 │ easy                     │
  ├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
  │ ruby 2.7.8 → 3.2.8                                        │ easy                     │
  ├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
  │ puma ~> 6.0 — Puma 5 refuses Rack 3                       │ moderate, well-known                 │
  ├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
  │ concurrent-ruby unpinned — pin only needed below 7.1      │ moderate (the Gemfile comment hints) │
  ├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
  │ deep_cloneable ~> 3.2 — 3.1 caps ActiveRecord < 7         │ hard, gemspec trivia                 │
  ├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
  │ liquid 3 → 4 — liquid 3 taint-checks, dead on Ruby 3.2    │ hard, obscure                     │
  ├───────────────────────────────────────────────────────────┼──────────────────────────────────────┤
  │ tinymce-rails 5.10.2 → 5.10.9 — 5.10.2 calls File.exists? │ very hard, deeptrivia               │
  └───────────────────────────────────────────────────────────┴──────────────────────────────────────┘