Prompt (identical across all runs):
What was the sushi place near latitude 41 in Columbus? Did it go out of business I don't see it on Google maps anymore
Ground truth: The place is Rishi Sushi Kitchen & Bar, 114 N. 3rd St., downtown Columbus — a few blocks from the Latitude 41 restaurant (50 N. 3rd St.). Rishi closed in January 2026 (decision Jan 10, announced Jan 13). The owners later announced a successor concept, SAGE, at the same address for summer 2026.
Runs: 15 total across 5 vendors — Meta (Llama 3.1 8B, Llama 3.3 70B), Anthropic (Haiku 4.5, Sonnet 4.6, Sonnet 5, Opus 4.6/4.7/4.8), OpenAI (GPT-5.5), and Zhipu (GLM-5.2, run agentically in Claude Code) — at varying effort levels.
Caveats on the numbers. Character counts are approximate magnitudes read off the transcripts — not tokenizer output, not byte-exact. The GLM row is the exception: those token figures are real, taken from the OpenTelemetry trace. "Turns to correct answer" is somewhat interpretive at the margins (see notes).
| Model / Effort | Turns (total) | Turns to correct answer | Tool calls | Tool types | Fetched full page? | Output chars (approx) | Tool text in (approx) | Named Rishi? | Identified closed? | Precision-beyond-retrieval |
|---|---|---|---|---|---|---|---|---|---|---|
| Llama 3.1 8B | 1 | — | 0 | none | no | ~600 | 0 | no (no tools) | no | none — no claims made |
| Llama 3.3 70B (Exa) | 1 | 1 | 1 | web_search_exa ×1 | read snippets | ~450 | ~5,000 | yes | yes | good date discipline — "earlier this year," no date; but ungrounded Oshio "closed" relevance-conflation |
| Haiku 4.5 | 1 | — | 2 | web_search ×2 | no | ~1,600 | ~3,500 | no (offered Sushi Time) | ~ wrong place, hedged | ungrounded Kura address |
| Sonnet 5 / med (fail) | 1 | — | 1 | conversation_search | no | ~900 | ~9,000 | no (bounced to user) | no | — |
| Sonnet 5 / med (recovery) | 5 | 4 (nudge + screenshots) | 1 | web_search | no | ~2,400 | ~4,500 | yes | no first, yes after screenshots | wrong date, user-corrected |
| Sonnet 4.6 / med | 1 | 1 | 3 | web_search ×3 | no | ~2,600 | ~5,500 | yes | yes | fabricated quotes; empty citations |
| Sonnet 4.6 / high | 1 | 1 | 2 | web_search ×2 | no | ~3,200 | ~4,000 | yes | yes | worst fabricated quotes; hardened "2014" |
| Sonnet 5 / high | 1 | 1 | 3 | places ×2, web ×1 | no | ~3,500 | ~40,000 | yes | yes | asserted "Jan 13"; 12-vs-15yr ignored |
| GPT-5.5 (ChatGPT) | 2 | 2 | 3 | search ×1, web.run ×2 | read snippets (t2) | ~2,800 | ~8,000 | t2 yes | t1 no, t2 yes | read before dating; stayed vague, didn't over-claim |
| Opus 4.6 / med | 1 | 1 | 3 | web_search ×3 | no | ~2,900 | ~6,500 | yes | yes | asserted "Jan 10"; mostly paraphrased |
| Opus 4.7 / med | 3 | 3 (2 nudges) | 3 | places ×2, web ×1 | no | ~2,900 | ~18,000 | yes | no first (wrong entity), yes after | invented "David Kim/Moshi"; "Jan 13/15yr" unfetched |
| Opus 4.7 / xhigh | 2 | 2 (1 clarify) | 3 | places ×2, web ×1 | no | ~3,000 | ~35,000 | yes | yes | confabulated 2-date reconciliation |
| Opus 4.8 / med | 3 | 3 (2 nudges) | 3 | places ×3, web ×1 | no | ~3,600 | ~55,000 | yes | no first (wrong entity), yes after | restrained; "Jan 10" + 2-date confab, no fetch |
| Opus 4.8 / high | 3 | 2 (1 clarify) | 4 | places ×3, web ×1, ask_user ×1 | no | ~4,200 | ~70,000 | yes | yes | invented "Best Sushi 2017"; 2-date confab |
| GLM-5.2 (Claude Code + Exa, agentic) | 1 | 1 | 5 | web_search_exa ×4, web_fetch_exa ×1 | YES | ~2,750 (visible; +~14,400 thinking) | ~39,000 | yes | yes | BEST — fetched, so "Jan 10 decided / Jan 13 announced" is grounded not confabulated; only lapse: unverified "Haiku 2016" as an aside |
GLM token counts (real, from trace): ~5,087 / 9,741 / 3,882 input and ~1,743 / 1,730 / 963 output across three GLM-5.2 turns; ~36K cache-read by final turn; ~131s total.
| Outcome | Count | Runs |
|---|---|---|
| Named Rishi (eventually) | 12 / 15 | Llama 70B, GLM-5.2, both Sonnet 4.6, Sonnet 5 high + med-recovery, Opus 4.6, both 4.7, both 4.8, GPT-5.5 (t2) |
| Failed to name Rishi | 3 / 15 | Llama 8B, Haiku, Sonnet 5 med-fail |
| Closure correct — first attempt | 8 / 15 | GLM-5.2, Llama 70B, both Sonnet 4.6, Sonnet 5 high, Opus 4.6 med, Opus 4.7 xhigh, Opus 4.8 high |
| Closure correct — only after user correction | 4 / 15 | Sonnet 5 med-recovery, GPT-5.5, Opus 4.7 med, Opus 4.8 med |
| Never resolved | 3 / 15 | Llama 8B, Haiku, Sonnet 5 med-fail |
| Fetched a full closure article before dating | 2 / 15 | GPT-5.5 (t2), GLM-5.2 |
| Avoided over-claiming the date | 3 / 15 | GPT-5.5 (read), Llama 70B (stayed vague), GLM-5.2 (read → grounded specifics) |
| Got latitude discrepancy right, first turn | 3 / 15 | Opus 4.7-xhigh, Opus 4.8, GLM-5.2 (only one to quantify + hypothesize misremembering pre-search) |
| Axis | Verdict | Fixed by? |
|---|---|---|
| Routing (world vs. memory on turn 1) | Biggest predictor of success | Higher effort |
| Parse / calibrated clarification (landmark-vs-entity; search→fail→ask, not guess) | Emerges with capability & effort | Higher effort |
| Precision-beyond-retrieval (claims exceeding what was read) | The invariant — held for every snippet-only run; broken only by fetching or by staying vague | Fetch-before-cite (demonstrated, not just inferred) |
1. Routing is the biggest lever, and it's effort-sensitive. Failing runs pattern-matched the "past tense + I don't see it anymore" phrasing to a memory-recall task (Sonnet 5 med reached for conversation_search) or asked the user to do the retrieval. Winning runs treated "Latitude 41" as a searchable entity and searched immediately. Every high/xhigh run nailed turn 1; medium was a coin-flip.
2. Parse-fails-at-medium replicates across both Opus generations. Opus 4.7 and 4.8 each solve the landmark-vs-entity parse cleanly at high/xhigh but misread it at medium — producing a confidently-wrong "still open" intermediate about the wrong entity (Latitude 41 itself), requiring a second user correction. Same model, lower effort, same failure, twice. Cleanest evidence that opening-move quality is effort-governed, not capability-governed.
3. "Total turns" hides why a run was slow. A 3-turn run can mean a deliberate, good clarification (Opus 4.8 high: 1 clarify, then solved) or a misfire the user had to correct (Opus 4.7/4.8 medium: 2 nudges after a wrong answer). The "turns to correct answer" column separates them. All the genuinely costly (3+ turn) cases are medium-effort.
4. Naming Rishi ≠ getting it right. 12 runs named it, but 2 first asserted the wrong operational status (Sonnet 5 med-recovery "still open"; GPT-5.5 t1 "not out of business"). Retrieval of the entity and retrieval of its status are separable.
5. Precision-beyond-retrieval was the invariant — and it now has a clean existence proof of the fix. Every run that answered from snippets over-claimed the closure date (fabricated "Jan 13," confabulated "decided Jan 10 / announced Jan 13" reconciliations, invented awards/owners). The invariant was broken by exactly the runs that did something different:
- GLM-5.2 (agentic, Claude Code + Exa) issued a real
web_fetch_exaon the closure article, read that it was "announced in a Jan. 13 post," and reported "decision made Jan 10, announced Jan 13" — the same two-date structure the Opus runs confabulated, but here grounded in the fetched text. Same sentence, opposite epistemic status. - GPT-5.5 turn 2 read snippet content before dating and stayed vague where sources conflicted.
- Llama 3.3 70B avoided a fabricated date by simply saying "earlier this year."
The single best answer in the set is not from the largest model — it's from the one (GLM-5.2) that did the cheap, boring, universally-skipped step of opening the article before writing the date down.
6. More retrieved context did not buy more grounding. Input size tracked tool choice, not effort: web_search returns lean snippets (~2–4K chars); places_search returns fat records (5 reviews + hours + coords each), so places-heavy runs pulled 35–70K chars. The ~70K-char run (Opus 4.8 high) produced the most fabrication — the fat context was operating-restaurant data, while the closure facts lived in article bodies no snippet-only run opened. Grounding tracked what was read, not how much.
7. The two open models bracket the small-model failure modes. Llama 8B: correct parse, zero agency (never searched). Llama 70B: full agency, read snippets, correct outcome, but no latitude engagement and one padded relevance-conflation (Oshio). Agency and comprehension are separable and scale independently.
Every capable run found Rishi. The unsolved problem was never retrieval — it's the discipline of not embroidering a found answer with retrieved-sounding specifics (dates, tenures, awards, quotes) the model never actually read. Effort and model scale reliably fix routing and mostly fix over-quoting; they do not fix — and can ornament — unfetched precision.
The fix is now demonstrated rather than hypothesized: the two runs that fetched-and-read (GLM-5.2, GPT-5.5) are the two that got the specifics grounded, and a third (Llama 70B) stayed safe by hedging. Fetch the source, read it, and make claims exactly as precise as the page supports — no more. That single step, skipped by all eight Claude runs at every effort level, is what separated the confident-and-wrong from the confident-and-right.