This work builds on Mike Lay’s amazing SessionBoard Eval Kit (sbek).
SBEK does the hard and important work: it defines the evaluation scenarios and rubrics, drives a browser through MCP, records evidence, separates browsing from fresh-context judging, validates judgements, and produces the final score. It is thoughtfully designed, unusually rigorous, and was a joy to build on.
The adapter described here does not replace SBEK and should not be mistaken for its original implementation. SBEK already documents a harness path for both Claude Code and Codex. Our narrower contribution is automating that path as an unattended fleet: one fresh codex exec process per worker and judge, selected behind a rollback-safe executor flag.
- Original author: Mike Lay (
mkly) - Patch target / upstream repository: github.com/mkly/killmysaas-evals-coding-agent
It’s been ages since I’ve run out of Opus tokens (only Fable!). Doing multiple eval loops exhausted my Claude model quota. So let’s use the Codex CLI subscription.
The question became: can we switch the fleet to codex exec without weakening SBEK’s core contracts?
Those contracts matter more than the choice of model:
- Every browser worker starts with fresh context.
- Workers use SBEK’s MCP browser tools and leave durable evidence.
- Every judge starts with fresh context, separate from the worker.
- Judge output is schema-valid JSON with every required rubric item.
- No metered API key can accidentally be used.
- Claude remains available as an immediate rollback.
The answer was yes.
Add one global selector and two optional role-specific overrides:
SBEK_EXECUTOR=codex # default for workers and judges
SBEK_WORKER_EXECUTOR=... # optional: codex or claude
SBEK_JUDGE_EXECUTOR=... # optional: codex or claudeThat supports three useful modes:
# New default: Codex workers and Codex judges
SBEK_EXECUTOR=codex ./run-fleet
# One-flag rollback: Claude workers and Claude judges
SBEK_EXECUTOR=claude ./run-fleet
# Mixed canary
SBEK_WORKER_EXECUTOR=codex \
SBEK_JUDGE_EXECUTOR=claude \
./run-fleetThe selected executor and model should be written into the run manifest. Otherwise, a small cross-model judgement difference can be mistaken for a product regression.
Each scenario gets a new, ephemeral Codex process. The SBEK MCP server is supplied inline, required, and restricted to the browser/evidence tools the worker needs:
env -u OPENAI_API_KEY -u CODEX_API_KEY \
codex exec \
--ephemeral \
--ignore-user-config \
--ignore-rules \
--skip-git-repo-check \
--model "$MODEL" \
--json \
-c 'mcp_servers.sbek.command="npx"' \
-c 'mcp_servers.sbek.args=["--no-install","tsx","src/mcp.ts"]' \
-c 'mcp_servers.sbek.required=true' \
-c 'mcp_servers.sbek.enabled_tools=["start_scenario","abort_scenario","navigate","snapshot","click","fill","select","drag","upload","press","scroll","wait","screenshot","observe","done"]' \
"$SCENARIO_PROMPT"Before launching, the wrapper checks that codex login status reports ChatGPT subscription authentication. Removing OPENAI_API_KEY and CODEX_API_KEY from the child environment makes accidental metered API use fail closed.
Use the strongest sandbox supported by your host. Do not blindly add a sandbox-bypass option from somebody else’s environment; if ordinary Codex sandboxing is unavailable, add OS-level containment and run against a disposable test target under an unprivileged account.
Each rubric area gets a separate ephemeral process. Codex receives the judgement JSON Schema and writes only its final structured answer to the judgement file:
env -u OPENAI_API_KEY -u CODEX_API_KEY \
codex exec \
--ephemeral \
--output-schema "$JUDGEMENT_SCHEMA" \
--output-last-message "$JUDGEMENT_FILE" \
--model "$MODEL" \
--json \
"$JUDGE_PROMPT"--ephemeral prevents session persistence. A new OS process per area prevents accidental resume. Separate test invocations produced separate thread IDs, confirming fresh contexts in practice.
Structured generation is necessary but not sufficient. Our first probe exposed a subtle trap: a judge that cannot read the evidence may still emit syntactically valid JSON containing an empty items array.
The durable fix was defense in depth:
- the JSON Schema requires at least one judgement item;
- the wrapper independently extracts the expected rubric IDs from
sbek judge-brief; - the generated IDs must match exactly, in order;
- verdict, confidence, severity, and field types are checked again before scoring.
In pseudocode:
expected = rubric IDs from sbek judge-brief
actual = item IDs from generated judgement
require items is non-empty
require actual == expected
require every enum and field type is valid
only then run sbek score
This guard benefits both Codex and Claude.
We tested the two contracts independently before changing the default:
- A Codex worker completed a real SBEK browser scenario and produced dozens of screenshots, observations, and MCP tool calls.
- A separate Codex judge consumed frozen evidence and produced JSON accepted by SBEK’s validator, with every required rubric item present.
- The resulting report was not withheld for insufficient coverage.
- Separate judge invocations had separate thread IDs.
- No API-key environment variable was available to either process.
- The Claude path remained intact and selectable with one flag.
The models did not produce byte-identical opinions, nor should anyone expect them to. “Deterministic” here means deterministic structure and complete rubric coverage—not identical semantic judgement. Record the executor in the manifest and compare score trends within the same judge model whenever possible.
The portable change is deliberately small:
- timeout "$TIMEOUT" claude -p "$PROMPT" --model opus ...
+ case "$EXECUTOR" in
+ codex) run_codex "$PROMPT" ;;
+ claude) run_claude "$PROMPT" ;;
+ esacThen keep all SBEK-specific planning, MCP behavior, evidence, consolidation, and scoring outside that branch. The adapter selects a process; it does not fork the evaluation semantics.
The fleet no longer has a hard dependency on available Opus quota. Codex is the default for both browser workers and fresh-context judges, while Claude remains a one-flag fallback.
The larger lesson is not “model A beats model B.” It is that model choice should be an adapter behind a stable evaluation contract. Mike Lay’s SBEK made that possible because its browser, evidence, judge, and scoring boundaries were already so clean.