Prompt abbreviation vs token count (o200k_base): pure prose saves ~50-66%, but prompts embedding code/JSON/SQL cap out at ~13-19%

tok2.py

import tiktoken

enc = tiktoken.get_encoding("o200k_base")


def n(s):
    return len(enc.encode(s))


# ---------------------------------------------------------------
# Shared incompressible payloads (must survive abbreviation intact)
# ---------------------------------------------------------------

PY_CODE = '''```python
def merge_intervals(intervals: list[tuple[int, int]]) -> list[tuple[int, int]]:
    if not intervals:
        return []
    intervals.sort(key=lambda iv: iv[0])
    merged = [intervals[0]]
    for start, end in intervals[1:]:
        last_start, last_end = merged[-1]
        if start <= last_end:
            merged[-1] = (last_start, max(last_end, end))
        else:
            merged.append((start, end))
    return merged


def busiest_overlap(intervals):
    events = []
    for s, e in intervals:
        events.append((s, 1))
        events.append((e, -1))
    events.sort()
    best = cur = 0
    for _, delta in events:
        cur += delta
        best = max(best, cur)
    return best
```'''

JSON_SCHEMA = '''```json
{
  "type": "object",
  "required": ["invoice_id", "issued_at", "line_items", "total"],
  "properties": {
    "invoice_id": {"type": "string", "pattern": "^INV-[0-9]{6}$"},
    "issued_at": {"type": "string", "format": "date-time"},
    "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
    "line_items": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["sku", "quantity", "unit_price"],
        "properties": {
          "sku": {"type": "string"},
          "quantity": {"type": "integer", "minimum": 1},
          "unit_price": {"type": "number", "exclusiveMinimum": 0},
          "discount_pct": {"type": "number", "minimum": 0, "maximum": 100}
        }
      }
    },
    "total": {"type": "number"}
  }
}
```'''

STACK_TRACE = '''```
Traceback (most recent call last):
  File "/app/services/billing/worker.py", line 214, in process_batch
    invoice = render_invoice(order, customer, tax_table)
  File "/app/services/billing/render.py", line 88, in render_invoice
    line_total = item.unit_price * item.quantity * (1 - item.discount_pct / 100)
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
```'''

MD_TABLE = '''| Endpoint | Method | Auth | Rate limit | Idempotent |
|---|---|---|---|---|
| /v2/invoices | POST | Bearer | 10/min | via Idempotency-Key |
| /v2/invoices/{id} | GET | Bearer | 120/min | yes |
| /v2/invoices/{id}/void | POST | Bearer + admin scope | 5/min | yes |
| /v2/webhooks | POST | HMAC signature | n/a | no |'''

SQL_QUERY = '''```sql
SELECT c.customer_id,
       c.region,
       DATE_TRUNC('month', o.created_at) AS month,
       SUM(oi.quantity * oi.unit_price)  AS revenue,
       COUNT(DISTINCT o.order_id)        AS orders
FROM customers c
JOIN orders o        ON o.customer_id = c.customer_id
JOIN order_items oi  ON oi.order_id = o.order_id
WHERE o.status NOT IN ('cancelled', 'refunded')
  AND o.created_at >= NOW() - INTERVAL '12 months'
GROUP BY 1, 2, 3
HAVING SUM(oi.quantity * oi.unit_price) > 1000
ORDER BY revenue DESC;
```'''

# ---------------------------------------------------------------
# Experiments: (category, variants dict, incompressible payloads)
# Each variants dict maps variant name -> full text.
# "Full" is always the baseline.
# ---------------------------------------------------------------

experiments = []

# 1. Pure prose, long-form (best case for abbreviation)
experiments.append(("Pure prose: meeting summary brief", {
    "Full": (
        "Please summarize the following meeting transcript. Focus on action items, "
        "decisions that were made, unresolved questions, and any deadlines that were "
        "mentioned. For each action item, identify the owner if one was assigned and "
        "note whether a due date was stated explicitly or merely implied. Where the "
        "discussion was inconclusive, briefly characterize the disagreement and list "
        "the participants on each side. Distinguish clearly between decisions that "
        "were finalized during the meeting and proposals that still require sign-off "
        "from someone who was not present. Finally, produce a short section of "
        "suggested agenda items for the follow-up meeting, ordered by urgency. "
        "Present all of the results as nested bulleted lists under clear headings, "
        "and keep the total length under four hundred words."
    ),
    "Abbreviated": (
        "pls sum mtg transcript. focus: actions (owner? due date explicit/implied), "
        "decisions, open qs, deadlines. inconclusive topics: note disagreement + who "
        "on each side. separate finalized decisions vs proposals needing absent "
        "sign-off. end w/ follow-up agenda ordered by urgency. nested bullets under "
        "headings, <400 words."
    ),
    "Custom shorthand": (
        "S mtg transcript: AI(owner,DD expl/impl) DEC OQ DL; inconcl->disagr+sides; "
        "final DEC vs pending-signoff; FU agenda by urgency; nested bullets+hdgs; <400w."
    ),
}, []))

# 2. Prose wrapping a code sample (code must stay intact)
experiments.append(("Code review request (Python)", {
    "Full": (
        "Please review the following Python module for correctness, efficiency, and "
        "readability. Pay particular attention to edge cases such as empty input, "
        "intervals that share a boundary point, and intervals that are fully "
        "contained inside another interval. Comment on whether the sorting strategy "
        "is optimal and whether the sweep-line approach in the second function "
        "handles touching intervals the way a caller would expect. Suggest concrete "
        "improvements with code where appropriate.\n\n" + PY_CODE
    ),
    "Abbreviated": (
        "review py module: correctness, perf, readability. edge cases: empty input, "
        "shared boundaries, fully-contained intervals. is sort optimal? does "
        "sweep-line handle touching intervals as expected? suggest fixes w/ code.\n\n"
        + PY_CODE
    ),
}, [PY_CODE]))

# 3. Structured-output prompt with a JSON Schema (schema untouchable)
experiments.append(("Extraction prompt with JSON Schema", {
    "Full": (
        "You will be given the raw text of a scanned invoice. Extract the fields "
        "described by the JSON Schema below and return a single JSON object that "
        "validates against it. If a field is missing from the document, omit it "
        "rather than guessing, unless it is listed as required, in which case use "
        "null and add the field name to a top-level list called \"missing\". Do not "
        "include any commentary outside the JSON object.\n\n" + JSON_SCHEMA
    ),
    "Abbreviated": (
        "extract fields per schema below from scanned invoice text -> single valid "
        "JSON obj. missing optional field: omit. missing required: null + add name "
        "to top-level \"missing\" list. JSON only, no commentary.\n\n" + JSON_SCHEMA
    ),
}, [JSON_SCHEMA]))

# 4. Debugging prompt with a stack trace (trace untouchable)
experiments.append(("Debugging prompt with stack trace", {
    "Full": (
        "Our nightly billing worker started failing after yesterday's deploy. The "
        "traceback below is representative of all the failures. Please explain the "
        "most likely root cause, identify which recent schema or code change could "
        "have introduced it, and propose both an immediate hotfix and a longer-term "
        "defensive fix, including where validation should live so this class of "
        "error cannot reach the worker again.\n\n" + STACK_TRACE
    ),
    "Abbreviated": (
        "nightly billing worker failing since yesterday's deploy, all failures like "
        "trace below. likely root cause? which schema/code change? give hotfix + "
        "long-term defensive fix + where validation should live.\n\n" + STACK_TRACE
    ),
}, [STACK_TRACE]))

# 5. API docs prompt with a markdown table (table untouchable)
experiments.append(("Docs rewrite with markdown table", {
    "Full": (
        "Rewrite the API reference section below for external developers. Keep the "
        "table exactly as it is, but add an introductory paragraph explaining the "
        "authentication model, a paragraph on how rate limits are enforced and what "
        "the client should do on a 429 response, and a short note on idempotency "
        "keys with a concrete example header. The tone should be friendly but "
        "precise, and every claim must be consistent with the table.\n\n" + MD_TABLE
    ),
    "Abbreviated": (
        "rewrite API ref for external devs. keep table verbatim. add: intro para on "
        "auth model, para on rate limits + client behavior on 429, note on "
        "idempotency keys w/ example header. friendly+precise, consistent w/ "
        "table.\n\n" + MD_TABLE
    ),
}, [MD_TABLE]))

# 6. SQL optimization prompt (query untouchable)
experiments.append(("SQL optimization request", {
    "Full": (
        "The analytics query below has become slow as the orders table has grown to "
        "roughly two hundred million rows. Please suggest indexing strategies, "
        "rewrite opportunities, and any partitioning scheme that would help, "
        "assuming PostgreSQL 16. Explain the reasoning behind each suggestion and "
        "estimate the relative impact. The query must continue to return exactly "
        "the same results.\n\n" + SQL_QUERY
    ),
    "Abbreviated": (
        "query below slow, orders ~200M rows, PG16. suggest indexes, rewrites, "
        "partitioning. explain reasoning + est. impact. results must stay "
        "identical.\n\n" + SQL_QUERY
    ),
}, [SQL_QUERY]))

# ---------------------------------------------------------------
# Report
# ---------------------------------------------------------------

grand = []
print(f"{'':62s} {'tokens':>7s} {'saved':>7s}")
for title, variants, payloads in experiments:
    base = n(variants["Full"])
    payload_tokens = sum(n(p) for p in payloads)
    print(f"\n== {title} ==")
    if payloads:
        prose = base - payload_tokens
        ceiling = prose / base
        print(f"   incompressible payload: {payload_tokens} tok "
              f"({payload_tokens/base:.0%} of prompt) -> max possible saving {ceiling:.0%}")
    for name, text in variants.items():
        t = n(text)
        saved = (base - t) / base
        print(f"   {name:58s} {t:7d} {saved:7.1%}")
        if name != "Full":
            grand.append((title, name, base, t, saved, payload_tokens))
    # verify payloads survived abbreviation intact
    for name, text in variants.items():
        for p in payloads:
            assert p in text, f"payload mangled in {title} / {name}"

# ---------------------------------------------------------------
# Aggregate: savings on prose portion only (isolating the effect)
# ---------------------------------------------------------------
print("\n== Savings on the compressible (prose) portion only ==")
for title, name, base, t, saved, payload in grand:
    if payload:
        prose_full = base - payload
        prose_abbr = t - payload
        print(f"   {title:45s} prose {prose_full:4d} -> {prose_abbr:4d} "
              f"({(prose_full-prose_abbr)/prose_full:.1%} of prose, "
              f"but only {saved:.1%} of whole prompt)")

token-abbreviation-results.md

Prompt abbreviation vs. token count (o200k_base)

An experiment measuring how many tokens you actually save by abbreviating prompts — including realistic prompts that embed code, JSON Schemas, stack traces, markdown tables, and SQL, where the payload must survive byte-identical and only the surrounding prose can be compressed.

Tokenizer: tiktoken, encoding o200k_base (GPT-4o and newer).

Results

Pure prose (best case)

Variant	Tokens	Saved
Full (meeting-summary brief)	141	—
Abbreviated (pls sum mtg transcript…)	66	53.2%
Custom shorthand (S mtg transcript: AI DEC OQ DL…)	48	66.0%

A shorter version of the same brief (36 tokens full) showed the same pattern: 47.2% and 63.9% saved — so abbreviation of pure prose scales well.

Mixed prompts with incompressible payloads

The embedded code/schema/trace/table is asserted byte-identical between the full and abbreviated variants — only the instructions around it were compressed.

Prompt	Full	Abbrev.	Saved	Payload share	Max possible
Code review request (Python module)	264	228	13.6%	70%	30%
Extraction prompt + JSON Schema	322	281	12.7%	74%	26%
Debugging prompt + stack trace	171	139	18.7%	58%	42%
Docs rewrite + markdown table	192	164	14.6%	58%	42%
SQL optimization request	211	179	15.2%	69%	31%

Savings on the compressible (prose) portion only

Prompt	Prose tokens	Prose saved	Whole-prompt saved
Code review request	78 → 42	46.2%	13.6%
JSON Schema extraction	83 → 42	49.4%	12.7%
Stack-trace debugging	71 → 39	45.1%	18.7%
Markdown-table docs	81 → 53	34.6%	14.6%
SQL optimization	66 → 34	48.5%	15.2%

Takeaways

1. Pure prose compresses well and scales. ~50% from plain abbreviation, ~65% with a custom shorthand, on both short and long briefs.
2. It's Amdahl's law for tokens. In realistic mixed prompts the incompressible payload is 58–74% of the token count. The prose still compresses at roughly the same rate (35–49%), but the whole-prompt saving lands at only 13–19%.
3. Structured payloads are token-dense. A ~25-line JSON Schema costs 239 tokens — quotes, punctuation, and indentation tokenize poorly. In code-heavy prompts, the real savings come from trimming the payload (minimal repro instead of the whole module, fewer schema fields, one representative trace), not from abbreviating the instructions.

Rule of thumb: shorthand pays off for prose-only system prompts and repeated instructions; for prompts wrapping code or data, expect ~15% at best — and weigh that against the clarity risk of terse instructions.

Follow-up: abbreviated agent prompts

A max-abbreviation prompt for a (fictional) PR-reviewer agent:

reviewer agent. handle psky PR in gh: gh CLI fetch PR, read diff+thread. co branch,
run tests, chk failing CI (caused vs flaky?). verify chgs match PR descr; chk
regressions in callers of mod fns. safe->approve; else req-chg w/ inline cmts
(file:line). diplomatic w/ author (prior rounds contentious). rpt: verdict |
blockers | nits.

Full-prose equivalent: 149 tokens → abbreviated: 89 tokens (40.3% saved). Slightly below the pure-prose ceiling because the prompt keeps structural elements (tool names, flag-like directives) that don't compress.

Two caveats for real agent use:

• Savings on a ~90-token prompt are dwarfed by the cost of one misread instruction downstream.
• Agent prompts are usually tiny next to what the agent then reads (the diff, the thread — the incompressible payload again), so whole-task savings are negligible. Abbreviation wins mostly on frequently repeated system prompts at scale.

Follow-up: output-side abbreviation (LLM writing, not reading)

Findings from testing whether an LLM can adopt the shorthand for its output just by being instructed to:

1. Instruction suffices — doing ≠ measuring. To save output tokens the model doesn't need to count them, just emit fewer; its output (including extended-thinking tokens) is by construction in its native tokenizer. What it cannot do without tooling is report how many tokens were saved.
2. No token introspection. LLMs process text in tokens but have no reliable meta-access to token boundaries (same failure family as "count the r's in strawberry"). Exact counts require running the real BPE: tiktoken for OpenAI encodings, the Anthropic count_tokens API for Claude. Char-based heuristics (~4 chars/token) fail precisely on abbreviation questions: "pls" and "please" are both 1 token, so a heuristic predicts savings that aren't real. Rare abbreviations can even tokenize worse than the full word.
3. Cross-tokenizer transfer is directional, not exact. Tokenizers trained on similar corpora share the broad pattern (common words = 1 token, punctuation-dense payloads expensive), so o200k_base results approximate other modern tokenizers — but only approximately.
4. Model-to-model channels are the best fit. When both ends are LLMs sharing context, abbreviated messages decompress reliably; good target for agent-to-agent traffic.
5. Don't compress the scratchpad. Reasoning quality partly rides on the redundancy of fully spelled-out steps — more compute per conclusion, fewer skipped steps. Compressing user-facing or agent-to-agent text is cheap; compressing the model's own chain-of-thought trades away exactly what those tokens were buying. Also, drift: models tend to revert to full prose in long outputs unless the abbreviation instruction is reinforced.