How to benchmark an MCP tool server against Claude Code's built-in tools, honestly: the instrument that works, four traps that silently corrupt the numbers, arm isolation, ground truth, and the limits. Corrections welcome in the comments.

benchmark-mcp-vs-builtin-tools-HOWTO.md

How to benchmark an MCP tool server against Claude Code's built-in tools — honestly

A reproducible recipe. The worked example compares skyline (a content-hash-guarded editing/search MCP server) against Claude Code's own Read/Grep/Glob/Edit/Bash, but nothing here is skyline-specific — swap in any MCP server.

Everything below was executed, not sketched. Where I got something wrong first, the wrong version is included, because the failure modes are the useful part.

If you think this could be done better: comment on this gist with a link to your own public gist. Corrections welcome, especially on the parts flagged as unresolved.

---

0. The question, stated so it can be falsified

Does an agent using the MCP server finish real tasks with fewer wrong answers, fewer tokens, and fewer round trips than the same agent using built-in tools?

Not "is the output smaller". A tool that returns 50 bytes and needs four extra calls is not cheaper. Correctness comes first: a tool that is cheap by being wrong is not cheap.

1. Four traps that will silently corrupt your numbers

Each of these cost me a run to find.

1.1 — Do not read tokens from the session transcript JSONL. ~/.claude/projects/<slug>/<session>.jsonl looks like the perfect instrument. Its usage.input_tokens is a streaming placeholder (literally 1 or 0 on most entries, never backfilled) and output_tokens excludes thinking. Only cache_read_input_tokens and cache_creation_input_tokens are trustworthy. See anthropics/claude-code#28197 (closed as not planned) — the reporter measured JSONL vs the real totals across ~1,300 requests/day and found input low by 174x and output by 17x. Every JSONL-based usage tool inherits this.

1.2 — If you do parse the JSONL (for exact result chars, which are fine), dedup by requestId. One API response containing N parallel tool calls is written as N separate assistant entries that repeat the same usage block verbatim. On one of my sessions: 238 assistant entries, 96 distinct requestId. Naive output_tokens sum 385,324; deduped 126,542 — 3.05x apart. And dedup last-wins, not first-wins: entries sharing a requestId can carry intermediate snapshots followed by a final one, so keeping the first undercounts (ccusage#888).

1.3 — Console OTel exporters emit nothing under -p --output-format json. The JSON output contract suppresses them. You need a real OTLP receiver. My first "telemetry" run produced zero records and looked like a config error.

1.4 — tool_source is documented but empty. The claude_code.tool_result event is supposed to carry tool_source: builtin | mcp | sdk_host_builtin_mcp, which would be a perfect arm discriminator. It was blank on every event I captured. Key off the tool-name prefix (mcp__<server>__*) and off which config the run used instead.

2. What to actually measure with

Two independent readouts that agree are worth more than one you trust.

A. The -p --output-format json envelope — zero infrastructure, per-run totals:

{ "num_turns": 7, "duration_api_ms": 21410, "total_cost_usd": 0.2216,
  "usage": { "input_tokens": 14, "output_tokens": 926,
             "cache_read_input_tokens": 304357, "cache_creation_input_tokens": 19394 } }

B. OpenTelemetry — per-call and per-request detail. Set on the CLI:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1     # required for traces
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_LOG_TOOL_DETAILS=1
export OTEL_LOGS_EXPORT_INTERVAL=1000            # default 5000; runs are short
export OTEL_METRIC_EXPORT_INTERVAL=2000          # default 60000 — you WILL lose data

I verified these agree exactly. Same run, two readouts:

	input	output	cache_read	cache_creation	cost
sum of 7 claude_code.api_request events	14	926	304,357	19,394	$0.2216
-p envelope	14	926	304,357	19,394	$0.2216

So input_tokens: 14 is real — it is the uncached remainder, with everything else served from cache. That is why trap 1.1 is about the JSONL writer specifically, not about the numbers themselves.

Useful fields: claude_code.api_request gives per-request tokens + cost_usd + duration_ms + request_id. claude_code.tool_result gives tool_name, tool_use_id, tool_result_size_bytes, duration_ms, success, error_type. Traces give claude_code.llm_request (ttft_ms, stop_reason) and split claude_code.tool.execution from claude_code.tool.blocked_on_user, so permission waits stop polluting tool latency.

Collector, in about two minutes

ColeMurray/claude-code-otel (MIT) is prewired for these exact signal names: collector + Prometheus + Loki + Grafana, make up (or docker compose up -d). Two edits are needed for benchmarking rather than dashboards — add a file exporter so raw records land on disk for deterministic per-run slicing, and a traces pipeline, which upstream does not have:

exporters:
  file:
    path: /etc/otel/out/records.json      # mount a writable volume for this
service:
  pipelines:
    metrics: { receivers: [otlp], processors: [resource], exporters: [prometheus, debug, file] }
    logs:    { receivers: [otlp], processors: [resource], exporters: [debug, otlphttp, file] }
    traces:  { receivers: [otlp], processors: [resource], exporters: [debug, file] }

Parsing footgun: the event name lives in body.stringValue as claude_code.<name>, while the event.name attribute holds the short name without the prefix. Match the attribute against claude_code.api_request and you silently get zero hits.

3. Arm isolation — the step that decides whether any of it means anything

An MCP server costs tokens whether or not it is called: its tool schemas sit in the context. So a baseline arm with the server "installed but unused" is not a baseline. Make it unreachable:

// arm-a.json  (built-ins only)
{ "mcpServers": {} }

// arm-b.json  (server under test only)
{ "mcpServers": { "skyline": { "type": "http", "url": "http://127.0.0.1:7333/mcp" } } }

claude -p --strict-mcp-config --mcp-config arm-a.json ...

--strict-mcp-config ignores every other MCP configuration — user scope, project scope, plugins. That also strips unrelated servers (Gmail, Drive, …) out of both arms, which you want.

Also neutralise your instruction files. My ~/.claude/CLAUDE.md tells every session to use the MCP server's tools. Arm A cannot reach them, so it burned turns hunting and even narrated it in the answer. Removing that contamination alone took arm A from 9 turns / $0.367 to 7 turns / $0.137 — 63% cheaper, same correct answer. Move the file aside for the run window and restore it in a finally.

--safe-mode disables CLAUDE.md, skills, plugins and hooks in one flag — but it disables MCP servers too, even with an explicit --mcp-config (measured: the arm-B probe replied NO_SKYLINE_TOOLS). Do not reach for it as a shared baseline.

4. Run protocol

• Pin the model AND the effort. An unpinned run inherits your saved default — mine silently ran at effort: low. --model sonnet --effort medium.
• Fresh session per run, own --session-id <uuid> that you generate: it is how you find the run's records later. Every OTel record carries session.id.
• One tool call per turn if you want per-call token attribution — usage is per API request, and a turn firing four calls cannot be split.
• Counterbalance arm order so drift cannot correlate with arm.
• ≥3 runs per cell, report median and range. Do not claim p95 off n=3.
• Measure COLD and WARM separately. A fresh session per task makes the MCP arm pay discovery and any first-call gate every task, which real use amortises. Run both: fresh session per task, and all tasks in one session via --resume <session-id>. The truth is between them, and the gap is itself a finding.

5. Ground truth, or you are grading vibes

Use known-answer tasks with mechanical scoring. The example corpus (filament-bench-corpus, public) ships bench/tasks.json: needle-based navigation tasks with an expect block, plus bench/verify_tasks.php which proves every needle still resolves uniquely at the current SHA. Run the verifier first — if it fails, the ground truth is stale and every number downstream is worthless.

Generate instructions from the manifest so nothing is hand-transcribed, and keep the wording tool-agnostic and identical across arms:

In <from.file> the expression `<from.needle>` refers to a declaration elsewhere in this repository. Find where that thing is declared. Reply with exactly one line in the form path/to/file.ext:LINE and nothing else.

Score by string match against the expected file and the line the expected needle sits on. Track three outcomes, not two:

outcome	why it matters
correct	file and line
right file, wrong line	partial credit, and it separates "found it" from "guessed near it"
wrong but confident	the expensive one — an answer that looks right and is not

Include deliberate traps. The example corpus has two same-named methods in different namespaces (App\Billing\Formatter::money vs App\Reporting\Formatter::money), so a tool that always answers "Billing" scores exactly 50% and an average would hide it. It also has a task whose correct answer is "this is unresolvable magic dispatch" — that needs its own scoring rule, so keep it out of a uniform template rather than mis-scoring it.

6. Report these, drop the rest

correct · output_tokens · turns · cost_usd · api_ms · cache_read and cache_creation kept separate.

Do not report: per-call bytes as a proxy for tokens (structured output tokenizes differently from prose); a blended cold+warm average; dashboard rate panels as if they were per-run totals; or the MCP server's own telemetry against an estimate of the built-in side.

7. Limits of this method — state them, do not bury them

• Tool discovery is harness-side. In my environment tools are loaded on demand through a ToolSearch step, which no setting disables. Both arms pay it, so it is symmetric, but the result measures the server in this harness, not the server in the abstract.
• The token gap is partly schema count, not verbosity. A server with 78 tools inflates context per turn regardless of what it returns. Report cache_creation/cache_read separately so a reader can see which it is.
• Prompt caching across serial runs is not resettable mid-session; interleave arms and report per-run numbers, not just aggregates.
• Cost is model- and effort-dependent. Publish the model id, the effort level, and the date. Mine: claude-sonnet-5, effort=medium, 2026-07.
• One task and one surface generalise to nothing. Say how many of each you ran.

8. Before you publish: scrub the telemetry

OTel records include user.email, user.id, user.account_uuid, user.account_id, organization.id, and (with OTEL_LOG_USER_PROMPTS=1 or OTEL_LOG_TOOL_DETAILS=1) prompt and tool content. OTEL_LOG_RAW_API_BODIES dumps entire request bodies. Strip those fields before putting records in a gist or a repo.

---

Do it better

The unresolved parts, in case you want to attack one:

1. Killing the harness's tool-discovery step so a true vanilla built-in baseline exists.
2. A defensible way to amortise MCP schema cost per task instead of per session.
3. Scoring rules for surfaces beyond single-answer locate — references, rename precision, "correctly refuses to answer".
4. Whether tool_source is populated in any version, which would make arm attribution automatic.

Comment below with a link to your own public gist if you benchmark this differently, and say what you changed and why. Numbers without a method are worth nothing; a method someone can attack is worth something.