The P100 has been doing silently noisy math in llama.cpp for years. Three lines fix it for free.
llama.cpp's CUDA backend has a FAST_FP16_AVAILABLE flag: "this GPU is fast at fp16, so do
quality-sensitive math in fp16." The GTX 10-series (sm_61) was exempted from it long ago. The
Tesla P100 (sm_60) never was — because GP100 is the one Pascal chip with fast fp16 hardware.
Hardware can, therefore software did. Nobody measured what it cost.
Measured against an fp32-arithmetic truth base (Qwen3.6-27B Q6_K, wikitext-2, 2048 ctx, 32 chunks, KLD over full logit distributions):
| median KLD vs truth | top-token agreement | |
|---|---|---|
| stock llama.cpp on P100 | 0.002298 | 96.53% |
| 3-line patch | 0.000001 | 99.89% |
That's ~2300× tighter, and roughly 1 in 29 next-token predictions were changing outright. The speed cost, benchmarked at pp8192/tg32 @ depth 8192 across three model classes (27B hybrid, 4B dense, 36B MoE): zero. Prefill inside noise on all three; decode +1.4% faster patched.
The patch is merged in llama-cpp-turboquant PR #212 and open in buun-llama-cpp PR #80. GGML upstream issue: ggml-org/llama.cpp#25593.
ggml/src/ggml-cuda/common.cuh has three gates that all say the same thing:
#if defined(FP16_AVAILABLE) && __CUDA_ARCH__ != 610 // device macro
ggml_cuda_highest_compiled_arch(cc) != 610 // fast_fp16_available()
cc >= GGML_CUDA_CC_PASCAL && cc != 610 // fast_fp16_hardware_available()sm_61 (GTX 1070/1080, P40) is carved out of all three — someone measured or knew that chip's
fp16 story and made the right call. sm_60 (P100) sailed through all three gates because GP100
has genuine 2:1 fp16 hardware (18.7 TF). The fix extends the existing sm_61 idiom:
&& __CUDA_ARCH__ != 600 / && cc != 600. Three lines. That's the whole patch.
What those gates control on sm_60 (verified in dispatch source, not headers):
- Flash-attention tile and vec kernels — all of attention math ran in half2 vector arithmetic. (Yes, llama.cpp has real working FA on Pascal — a fact the Python world, where FA2 requires sm_80, mostly doesn't know.)
- cuBLAS compute type — and this one's subtle: on sm_60, MMQ (the int8 quantized matmul
path) is hard-disabled (
should_use_mmqreturns false below the DP4A gate — GP100 missed the DP4A instruction by one minor arch version). So ALL quantized-weight prefill runs dequantize + cuBLAS GEMM, and this flag chose fp16 GEMM. Your entire weight path, in half precision, silently. - mmvf — f16-weight matrix-vector arithmetic type.
Not by reading code — by measuring GPUs against each other. While validating a KV-cache codec (buun's TCQ/VBR work) across a P100 rig and a 3090, the same model, same weights, same test corpus kept producing systematically different KLD floors. Chasing that difference through a controlled decomposition panel (KV storage precision × attention arithmetic × weight-path arithmetic, each isolated) pointed at platform arithmetic, not the codec. The codec was innocent. The platform wasn't.
Key methodology point for anyone reproducing this class of bug: within-build comparisons cannot see it. If your truth base and your test run share the same fp16 arithmetic, the error is common-mode and cancels. Every rung of a KV-quant ladder measured on stock sm_60 looks clean relative to its own f16 baseline while the whole ladder sits ~0.005 median KLD away from the fp32 answer (with a 95.0% same-top floor — the weight-path error is ~2× the attention-path error). You must generate the reference logits with fp32 arithmetic to see it.
- sm_60: measured, fixed. sm_61: was already exempt (nothing changes).
- sm_62 (Jetson TX2): same gate, same silicon family, unmeasured — deliberately NOT patched. Measure before you carve.
- Everything Volta+: unaffected by this patch. Independently verified by the turboquant maintainer pre-merge: these three gates are the only 600-vs-610 distinction in the CUDA tree, and a Blackwell build produced bit-identical PPL. (Whether other arches have their own distance-to-fp32-truth is a separate, open research question. Different flag, different kernels, different measurement. Don't extrapolate this bug to your 3090.)
P100s are flooding the used market at ~$80 shipped while the DRAM crisis prices everything else into orbit. That's 16GB of HBM2 at 732 GB/s per card. The market had it half-right: sm_61 P40s (clean math all along) soared to ~$300 while the P100 floundered — the price gap was partly pricing in a software bug. Post-patch, the P100 does fp32-clean inference at fp32 speed that was always there; the "fast" fp16 path it loses was buying nothing (real prefill is bound by cuBLAS GEMM and memory bandwidth, not the fp16 vector path).
- Build stock llama.cpp,
-DCMAKE_CUDA_ARCHITECTURES=60. - Generate truth base:
llama-perplexity --kl-divergence-base out.kldwith fp32 KV, FA off, on a patched build (fp32 arithmetic) — this is the step everyone skips. - Score stock vs patched runs against that base with
--kl-divergence, 2048 ctx, 32 chunks. - Speed:
llama-benchpp8192/tg32 @ d8192, patched vs unpatched, same host, nothing else on the GPUs.
Full panel design, predictions-logged-before-runs, and receipts: [https://github.com/apollo-mg/Project-Apollo/blob/1a41c11439e6661c84c83bec333ccdf964663da6/data/Apollo%20Docs/Pascal_FAST_FP16_Carveout_Results.md] [https://github.com/apollo-mg/Project-Apollo/tree/main/data/carveout_panel].
- buun (TCQ/VBR codec) — the cross-GPU codec validation that surfaced the discrepancy, and the first fork PR review.
- TheTom (llama-cpp-turboquant) — fast merge with independent cross-arch verification.
- Found and isolated by running Fable 5 through my custom P/ReAct/R agent loop. It wrote the scripts, the hardware provided the receipts.