apollo-mg

Llama-Server is Throwing Away Your Perfectly Good KV Caches, and How to Fix It

Introduction

I watched 2.49 GB of state restore from disk in 1.23 seconds — and then get thrown away. llama-server's slot save/restore promises exactly what long-context work on budget hardware needs: park a session on disk, bring it back later without paying the prefill tax again. And the restore itself works perfectly. But across a process restart, the feature was functionally useless: the first query after restoring discarded the entire rehydrated state and re-prefilled from scratch. The reason turned out to be a single piece of metadata that lived only in process memory.

Testbed: Qwopus3.6-27B Q6_K (hybrid architecture), 2× Tesla P100 layer-split, turbo4 KV quantization.

The Discovery

apollo-mg

The P100 has been doing silently noisy math in llama.cpp for years. Three lines fix it for free.

---

TL;DR

llama.cpp's CUDA backend has a FAST_FP16_AVAILABLE flag: "this GPU is fast at fp16, so do quality-sensitive math in fp16." The GTX 10-series (sm_61) was exempted from it long ago. The Tesla P100 (sm_60) never was — because GP100 is the one Pascal chip with fast fp16 hardware. Hardware can, therefore software did. Nobody measured what it cost.