https://claude.ai/code/artifact/1562ee2f-d12b-41d2-9dcd-686f4a614c6b
One-shot eval · self-hosted vs hosted Elevator Gauntlet Four one-shot generations of the same elevator-simulation prompt, scored on requirements, traced correctness, animation, and code quality — plus headless runtime testing with screenshots. The question: is self-hosted DeepSeek good enough for real agentic work?
Task single-file HTML/CSS/JS elevator sim Harness Claude Code (identical for all) Self-hosted stack GH200 · llama-server · UD-Q8_K_XL Date 2026-08-12 Verdict: at its best, DeepSeek-V4-Flash matches Sonnet 5 on this task. At its worst, it shipped the only broken build of the four. The gap between hosted and self-hosted here isn't capability — it's consistency.
Opus 5 produced the strongest build (33/40, "close to a reference implementation"). DeepSeek's interactive run (32/40) beat Sonnet 5 (30/40). But DeepSeek's second run — same model, same server, same config — scored 20/40 with two fatal logic bugs. Both Claude builds worked on every dimension tested. For real work: self-hosted DeepSeek is usable with a verification loop, not on one-shot trust.
Scoreboard Static scores are from four independent code-review agents tracing the actual dispatch and queueing logic, each scoring 1–10 per section. Runtime status is from a headless Playwright harness: load, spawn 6 people, watch 8 seconds, hover for tooltips.
Sample Requirements Correctness Animation Code quality Total Runtime Opus 5 hosted · self-tested with Playwright 9 7 9 8 33/40 ✓ works DeepSeek V4-Flash self-hosted · interactive claude-local run 9 7 8 8 32/40 ✓ works Sonnet 5 hosted 9 6 7 8 30/40 ✓ works DeepSeek V4-Flash self-hosted · scripted batch run, 26 min 6 2 6 6 20/40 ✗ frozen Runtime sanity — what actually happened in a browser Headless Chromium, 6 people spawned, element positions sampled 5 seconds apart. Movement counts are elements that changed position — the difference between a live simulation and a screenshot pretending to be one.
Sample JS errors Elements moved Tooltip Delivered anyone? Opus 5 0 28 / 60 ✓ works CSS ::after — DOM probe false-negative, verified in review ✓ done: 2 DeepSeek interactive 0 86 / 107 ✓ works ✓ delivered: 3 Sonnet 5 0 56 / 67 ✓ works ✓ delivered: 2 DeepSeek batch 0 4 / 77 ✓ works ✗ delivered: 0 Opus 5 sim after 8 seconds: rider visible inside car, counters live Opus 5 — rider visibly inside the car mid-flight; waiting 3 · riding 1 · done 2. People are CSS stick figures queued beside the shafts. DeepSeek interactive sim after 8 seconds: riders in two cars, deliveries counted DeepSeek (interactive) — riders in E1 and E3; waiting 6 · riding 1 · delivered 3. Clean queueing along each floor. Sonnet 5 sim after 8 seconds: cars moving between floors, deliveries counted Sonnet 5 — cars mid-flight with per-car destination labels; waiting 2 · delivered 2. DeepSeek batch sim after 8 seconds: all cars parked at bottom, nothing delivered DeepSeek (batch) — the failure, visible: all three cabs dead at floor 1, a person standing on a cab roof, shafts misaligned. Waiting 12 · riding 0 · delivered 0, forever. Per-sample review Opus 5 33/40 Best thing JS↔CSS timing contract: DOOR_MS=380 / WALK_MS=480 are the same numbers as the CSS transitions, each wait padded 40–60ms — animation and logic can't drift. Every one of six awaits in serve() is followed by an aligned alive() cancellation guard, and the author correctly narrows the guard after the rider leaves the roster — getting that wrong would have frozen the sim. Worst defect release() has no ownership check. Reset while cars are moving, then spawn: a zombie coroutine frees an occupied car and it gets double-booked — two riders in one 70px cab. Reviewer's line "Short of that, this is close to a reference implementation of the brief." DeepSeek — interactive run 32/40 Best thing Rider–car motion sync across separate DOM subtrees using a positionally-matched multi-value transition-duration — the rider glides with the car with no parenting and no rAF loop. Plus a bob keyframe deliberately written in margins to avoid clobbering transform centering. "Not things that survive by accident." Worst defect restack() fires the instant a person is dispatched — the next waiter is drawn on top of the departing one for 0.6–3.7s on every multi-person pickup. Stats also drop dispatched people from all counters for up to 4.35s. Both are several-line fixes. Reviewer's line "A strong 8/10 one-shot — the hard parts (async dispatch correctness, motion synchronization) are right." Sonnet 5 30/40 Best thing Promise-wrapped animation pipeline: the whole pickup-and-deliver sequence reads as fifteen linear awaits, with busy=true set synchronously before the first await so the capacity invariant can't be double-booked. Travel time scaled by distance, dest≠origin guarded, 0-index/1-display handled everywhere. Worst defect resetAll() doesn't cancel in-flight journeys. One click mid-run leaves ghost people, phantom deliveries, and eventually two coroutines fighting over one car — a cascading capacity violation with no self-healing. Reviewer's line "Solid, shippable-after-one-fix work whose flaws cluster where the model didn't picture the user's clicks." DeepSeek — batch run 20/40 Best thing The one-person capacity invariant is airtight — a scalar rider slot plus state re-validation closes even the race its own dispatcher creates. The visual layer (palette, CSS-drawn people, eased distance-proportional movement) is genuinely good. Worst defect ev.busy is a one-way latch — set on first move, cleared only by the Clear All button. Every elevator moves exactly once, then dies. Delivered is structurally pinned at 0. A second bug (no else for "person already on my floor") deadlocks ~60% of page loads before anything moves at all. Reviewer's line "It would score well in a screenshot and fails in about ten seconds of actual use." Findings that generalize Variance is the story, not capability. The same self-hosted model, same server, same flags produced a 32/40 and a 20/40 back to back. Hosted Claude produced two working builds. One-shot output from the self-hosted stack needs a run-verify-retry loop before you trust it. All three working builds share the same bug class. Sonnet, Opus, and DeepSeek-interactive each have an uncancelled-coroutine race on Reset. Every model reasoned carefully about the happy path and skipped lifecycle cancellation. If you want to find a bug in one-shot LLM code, click Reset while things are moving. Screenshot-quality ≠ working. The broken build has a good palette, hand-drawn CSS people, and eased animation. Static review caught it (2/10 correctness) and runtime testing confirmed it (4 elements moved vs 86). Both checks earn their keep; neither is sufficient alone. Opus's self-testing habit is the differentiator. It tested its own output with Playwright during generation — which is exactly the verification loop the self-hosted model needs imposed from outside. The capability gap is partly a workflow gap. Economics The batch generation took 26 minutes end-to-end (~10.5k tokens with thinking, sharing the GPU with a live session) ≈ $0.87 of GH200 time at $2/hr. Tuned single-session throughput: 36 tok/s decode (-ncmoe 24), ~620 tok/s prefill on Claude Code's ~25k-token system prompt (~40s to first token per call). Two concurrent sessions: ~23 tok/s each, ~47 tok/s aggregate. Reference: 2×H100 ≈ 30 tok/s at ~3× the price; A10/A100 ≈ 15 tok/s and unusable. Good, or great? Good — genuinely. The best DeepSeek build out-scored Sonnet 5 on this task, with animation-engineering details the reviewers called impossible to fake. That's not "impressive for a local model," it's simply competitive work.
But not great, and the reason is precise: greatness here isn't peak quality — it's the floor. Opus's floor was a reference-grade build; DeepSeek's floor was a sim frozen on load in 60% of page loads. Until the self-hosted stack wraps its output in the verification loop Opus runs on its own, treat it as a strong drafting engine whose work is checked before it ships — which, at $2/hr against $200/mo, is still a very good trade.
Four samples · identical prompt & harness (Claude Code 2.1.228) · static review by four independent tracing agents · runtime via headless Chromium (Playwright), 6 spawns, 8s observation · screenshots embedded above are the actual test frames · GH200 stack: llama.cpp + claude-code-proxy + DeepSeek-V4-Flash-0731 UD-Q8_K_XL (lossless)