Skip to content

Instantly share code, notes, and snippets.

@bgauryy

bgauryy/Headroom.md

Created June 8, 2026 14:16
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bgauryy/602fb0297e8a89dd016fa6485f216223 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bgauryy/602fb0297e8a89dd016fa6485f216223 to your computer and use it in GitHub Desktop.
Download ZIP
Explains how Headroom works under the hood
Raw
Headroom.md

Headroom Token Savings

A concise explanation of how Headroom reduces LLM token usage.

Summary

Headroom sits between an app/agent and the LLM provider. It rewrites large inputs into smaller, task-preserving forms before the request is sent. The original content is kept locally and can be retrieved later by hash.

It saves tokens through three main ideas:

  1. Content-aware compression — compress JSON, code, logs, diffs, HTML, search results, and prose with different strategies.
  2. CCR: Compress-Cache-Retrieve — send a compressed version now, store the original locally, and let the model retrieve the full content only if needed.
  3. Cache stability — detect volatile prompt data that can break provider prompt/KV caches.

How it is used

Headroom exposes several entry points:

  • Library/SDK mode — call compress(messages) or wrap an LLM client.
  • Proxy mode — run Headroom as a local proxy and point OpenAI/Anthropic-compatible clients at it.
  • MCP mode — use tools such as headroom_compress, headroom_retrieve, and headroom_stats.
  • Agent wrapping — wrap tools like Claude Code, Codex, Cursor, Aider, or Copilot.

All modes feed into the same compression pipeline.

Core pipeline

The Python runtime builds a transform pipeline:

messages
  -> CacheAligner
  -> ContentRouter
  -> compressed messages
  -> LLM provider

Current code says older message-dropping / rolling-window history mutation was retired. The active strategy is compression of eligible live-zone content, not blindly deleting chat history.

Technique 1: Content-aware routing

ContentRouter detects what each message contains and chooses a compressor:

Content type Strategy How it compresses Data-loss risk
JSON arrays / structured tool output SmartCrusher Keeps representative / relevant rows, schema shape, key fields, and summary-like structure instead of the full array. Falls back to Kompress or log compression if it does not shrink. Medium. Individual rows/items may be omitted from the prompt, but CCR can keep the original retrievable by hash.
Source code Code-aware compressor Uses code-aware structure instead of raw text, preserving important symbols, signatures, imports, and relevant blocks while trimming less useful code body/detail. Can fall back to Kompress. Medium to high if the exact omitted code matters. Headroom protects recent code and analysis-context code to reduce this risk.
Plain text / prose Kompress ML compressor Produces a shorter semantic compression of the text, keeping likely important meaning and dropping redundant wording/detail. Medium. This is lossy summarization/compression; exact phrasing and minor details can disappear.
Logs / build output Log compressor Keeps errors, warnings, unusual lines, and useful context while dropping repetitive noise and repeated boilerplate. Low to medium. Repeated lines are usually safe to drop, but a rare important line can be missed if classified as noise.
Search results Search compressor Keeps top/relevant matches, paths, snippets, and hit context instead of every returned result. Medium. Lower-ranked or seemingly unrelated hits can be omitted.
Git diffs Diff compressor Keeps changed hunks and important context while reducing surrounding unchanged lines and verbose patch detail. Medium. Exact full patch context may be lost unless retrieved.
HTML HTML extractor Extracts meaningful visible text/content and removes markup, scripts, styles, navigation, and boilerplate. Low to medium. Layout, attributes, hidden content, and scripts/styles are intentionally discarded.
Mixed content Section splitter + per-section strategy Splits input into sections, detects each section type, compresses each with the matching strategy, then joins the compressed sections. Varies by section. Risk is the sum of the selected strategies.

These strategies are mostly lossy in the prompt: they intentionally remove or rewrite detail to reduce tokens. CCR reduces permanent data-loss risk by storing originals locally and exposing retrieval by hash, but the model only sees the compressed form unless it asks for the original.

The router does not compress everything. It protects:

  • recent code,
  • user messages by default,
  • system/developer messages by default,
  • small content,
  • already-compressed CCR markers,
  • frozen prefix-cache messages.

For eligible content, it compresses cache misses in parallel, then replaces the original message content with the compressed string only when the result is smaller enough.

Technique 2: CCR — Compress-Cache-Retrieve

CCR makes aggressive compression safer:

  1. Compress a large tool output or document.
  2. Store the original locally with a hash.
  3. Send the compressed content plus a retrieval note/hash to the LLM.
  4. If the LLM needs the full data, it calls the retrieval tool with the hash.

This means the initial request can be much smaller without permanently losing information.

Technique 3: Compression caching

Headroom keeps a content-addressed cache of compressed results. If the same content appears again, it can reuse the compressed output instead of recompressing. It also tracks saved-token metadata for cached entries.

Technique 4: Provider cache awareness

CacheAligner is currently detector-only. It does not rewrite the prompt. It detects volatile content such as UUIDs, timestamps, JWT-shaped strings, and hashes in cache-hot prompt areas, then emits warnings/metrics. The goal is to avoid unstable prefixes that would reduce provider prompt-cache hits.

How savings are calculated

The accounting is straightforward:

tokens_saved = tokens_before - tokens_after

The pipeline:

  1. counts tokens before compression,
  2. applies transforms,
  3. counts tokens after compression,
  4. reports the delta.

CCR/MCP uses the same idea and returns:

tokens_saved = max(0, original_tokens - compressed_tokens)

Rust live-zone code uses the same formula per compressed block and sums the results.

Important nuance

Headroom's token savings are not magic and not guaranteed for every input. If content is already short, already compressed, recent/high-value, or not smaller after compression, Headroom may pass it through unchanged.

Source references

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment