Stealth_Bench_V1 — Run Analysis (browser-use + Steel + claude-opus-4-8)

Run: Stealth_Bench_V1 · framework browser-use (installed) · browser steel · model claude-opus-4-8 Started: 2026-06-19 18:39:20 (local) · Duration: ~6.4 h (23,066 s) · Tasks: 80 completed Headline: 52 pass / 28 fail (65%). ~93% of failures are bot-detection, not infra.

Run data: full per-task traces and the summary JSON are published at https://github.com/steel-experiments/bu-bench-results.git — see run_data/Stealth_Bench_V1_framework_browser-use_browser_steel_model_claude-opus-4-8_start_at_20260619_183920/ (80 task .json traces) and results/Stealth_Bench_V1_framework_browser-use_browser_steel_model_claude-opus-4-8.json.

TL;DR

The run-completion summary attributed the dominant failures to Steel/CDP/browser-side errors (-32001 Session not found, detached targets, missing CDP contexts, watchdog timeouts, reload loops) with captcha as one item. The persisted traces say the opposite:

26 of 28 failures are bot-detection / access walls the Steel session could not pass.
The CDP/Steel errors fired (visible in process logs and consistent with the recent bba0bdf / cfd11a8 / 809f3f2 recovery commits) but were absorbed by retry and are non-causal for pass/fail — they appear 0 times in the recorded task outcomes, including in the failing tasks.
The only true infra crash was 1 task (Steel API 502 Bad Gateway).

The lever that moves the pass rate is the browser provider's stealth quality, not the agent, the model, or CDP hardening.

What actually caused the 28 failures

Reconstructed from the 80 per-task trace files (judgement.reached_captcha, judgement.failure_reason, judgement.impossible_task):

Failure cause	Count	Evidence
Bot-detection / CAPTCHA walls	25	`reached_captcha=True`; failure_reason names Cloudflare / PerimeterX "Press & Hold" / DataDome / hCaptcha / slider / shape-match
Access-control wall	1	t70 — "Access denied" page (WAF/geo, antibot-adjacent; `reached_captcha=False`)
Auth/login wall (genuine feasibility)	1	t75 — Temu sign-in wall, `impossible_task=True` (needs credentials, not a bot issue)
Infra crash (judge never ran)	1	t29 — Steel API `502 Bad Gateway` at session create

So: 26 bot-detection/access walls, 1 real auth wall, 1 infra crash.

Bot-detection vendors seen this run: Cloudflare, PerimeterX ("Press & Hold"), DataDome, hCaptcha, slider and shape-match CAPTCHAs, plus generic "Access denied" / "You don't have access to this page" walls.

Failing task list

task	steps	dur (s)	captcha	impossible	reason
3	36	915	✓		Cloudflare on att…
9	4	75	✓		Cloudflare on entry
11	18	444	✓		Cloudflare "Verify you are human"
20	21	416	✓		Cloudflare on science.org
29	—	0	—	—	Steel API 502 Bad Gateway (crash)
30	9	291	✓		"Press & Hold"
31	12	199	✓		"Press & Hold" on Pricing
32	8	179	✓		"Press & Hold"
36	10	281	✓		"Press & Hold" on belk.com
37	5	85	✓	✓	PerimeterX on anntaylor.com
38	8	107	✓	✓	"Press & Hold" on entry
39	22	460	✓		"Press & Hold" after search
40	9	158	✓		PerimeterX "Press & Hold"
41	8	189	✓		"Press & Hold" on store locator
43	11	253	✓		"Press & Hold"
46	6	131	✓		"Press & Hold" on samsclub
49	6	127	✓		Yelp DataDome after first search
54	4	71	✓		DataDome on leboncoin.fr
55	10	184	✓		DataDome on entry
59	5	82	✓		antibot security check
61	8	173	✓		hCaptcha on entry
63	16	441	✓		slider CAPTCHA (Barcelona properties)
65	10	244	✓		human-verification captcha
70	14	347			"Access denied" page
72	13	382	✓		"You don't have access to this page"
75	7	165		✓	Temu sign-in wall (auth)
76	6	83	✓	✓	lululemon antibot ("Bad…")
79	21	604	✓		shape-matching CAPTCHA after search

CDP/Steel errors were recovered noise

Text-field scan of all 80 traces (agent steps, final result, judgement, error):

signature                       in PASS   in FAIL
session_not_found/-32001             0        0
detached/missing target              0        0
cdp_context_missing                  0        0
watchdog/screenshot/dom timeout      2        1

-32001, detached targets, missing CDP contexts: zero occurrences in task outcomes — including the failures. They were handled at the provider/session layer (logged to stderr, below the persisted agent_steps) and recovered by retry. They did not determine pass/fail. The CDP-hardening commits are good hygiene, but will not move the pass rate.

Nuance: bot detection is not 100% fatal. 3 passing tasks (2, 62, 64) reached a captcha and still passed — Steel bypassed detection there. That is roughly 1 captcha survival per ~9 captcha encounters (~11%); raising that rate is the win.

The two "Anthropic" observations are different failure classes

These are not the same bug.

ANTHROPIC_TOOL_USE_RETRY_SAVE_TEXT captured nothing because it instruments "Expected tool use in response but none found" — the API returning no tool_use block. That happened 0 times this run. The 809f3f2 recovery is working; Opus 4.8 emitted no empty-tool responses.
The Prologis anomaly was a different class. Task 66 (Prologis) passed (32 steps, 1262 s — the long tail is the retries) and its trace carries AgentOutput + PARAMETER_NAME markers. That is a Pydantic validation failure on browser-use's AgentOutput schema (model emitted a top-level placeholder instead of a well-formed action), not an empty-tool-use API error.

So SAVE_TEXT correctly captured nothing because the one anomaly that did occur is not the failure mode it watches for. To get visibility into the Prologis-class anomaly, instrument the AgentOutput validation branch, not the empty-tool-use branch. (This is the _content_block_diagnostics / ANTHROPIC_TOOL_USE_RETRY_SAVE_TEXT path added in browser_patches.py.)

The anomaly recovered via retry and did not persist as a failed-output diagnostic, matching the run summary.

Token & duration economics

           tasks   agent_tok   judge_tok   steps   duration
PASS          52   12,516,115     387,768     611    15,981 s
FAIL          28    5,989,668     165,680     307     7,086 s

Totals reconcile to the run header: 18.5 M agent tokens, 553 K judge tokens, 918 steps.

Failures cost ~32% of total agent tokens (6.0 M / 18.5 M) for zero yield.
Captcha failures look "short" (median 9 steps, 189 s) but still burned ~5.6 M tokens (~222 K each) because every step ships large DOM/screenshot payloads into Opus's context.
Worst offender: t3 — a captcha wall where the agent flailed 36 steps / 792 K tokens / 915 s before giving up. That single task is ~4% of the entire run's agent spend.
Top token burners: t3 (fail/captcha, 792K), t19 (pass, 631K), t66/Prologis (pass, 614K), t68 (pass, 600K), t35 (pass, 564K), t79 (fail/captcha, 501K). Most high-spend tasks are legitimately hard passes; the captcha failures (t3, t79, t39) are the pure waste.

Cost of the run

Priced from the exact token breakdown (the run recorded total_cost = 0.0, so it did not self-price) at Opus 4.8 $5/M input, $25/M output, with standard Anthropic cache rates (cache read 0.1× = $0.50/M, cache write 1.25× = $6.25/M).

Grand total ≈ $52 (~$49 if the judge runs on a cheaper model).

Agent (claude-opus-4-8) — $48.32

Token type	Tokens	Rate	Cost
Fresh input	6,099,217	$5.00/M	$30.50
Cache read	11,922,196	$0.50/M	$5.96
Cache write	13,222	$6.25/M	$0.08
Output	471,148	$25.00/M	$11.78
Agent total	18,505,783		$48.32

Judge — $4.03 (if also Opus 4.8)

Token type	Tokens	Cost
Fresh input	489,144	$2.45
Output	63,385	$1.58
Judge total	553,448	$4.03

The data records by_model: {} for the judge, so its model is unattributed. On a cheaper model (Sonnet/Haiku) the judge cost drops well under $1.

What drives it:

Brutally input-heavy: 18M input vs 0.47M output (38:1). Output is only $11.78; input dominates.
Prompt caching more than halved the cost. 66% of prompt tokens were cache hits. Without caching, 18.0M input tokens at $5/M = $90.17; with caching the input-side bill is $36.54 — a ~$54 saving (the agent alone would cost ~$102 instead of ~$48).
Fresh input is the biggest line ($30.50), because the cache invalidates whenever the DOM/screenshots change between steps.
Per task: ~$0.65. Per successful task: ~$1.00 (52 passes). The 28 failures burned ~$16 (≈32% of spend) for zero yield; task 3 alone was ~$2.

Sensitivity: the total lands in a $49–$61 band across plausible cache rates and judge models. The two numbers that matter are $5 input and $25 output.

Note on the summary file

results/Stealth_Bench_V1_framework_browser-use_browser_steel_model_claude-opus-4-8.json is a list of 2 run-groups, not a single run:

group 0 — leftover 20260619_182621 partial run (3 tasks, 1 pass / 2 fail).
group 1 — the full 20260619_183920 run (80 tasks, 52 / 28) — this is the run.

Report from group 1. Naively summing across groups yields 83 completed / 53 pass / 30 fail, which is inflated by the stale partial run.

Recommendations

The only lever that moves pass/fail is the browser provider's stealth quality. ~93% of failures are bot-detection. Investigate Steel's residential-proxy / anti-detect tier, or wire in a captcha solver. CDP hardening and agent/model changes will not move this number.
Segment results by reached_captcha / impossible_task when reporting. The judge already records these — use them to separate "agent couldn't do it" from "environment blocked it." Otherwise Opus 4.8 + browser-use read ~20 pts worse than they are.
Early-abort on detected captcha to cap wasted spend. t3 alone burned 792 K tokens retrying a wall it cannot pass. Detecting the captcha signal mid-run (the judge already knows it post-hoc) and aborting would have saved ~5 M tokens this run.
Instrument the AgentOutput validation branch for Prologis-class anomalies — SAVE_TEXT won't see them by design.

Generated from the 80 per-task trace files in steel-experiments/bu-bench-results.

nibzard/run_report.md

Select an option

No results found