Synthetic Agentic Benchmark Generation — Ontology

ontology.md

Synthetic Agentic Benchmark Generation — Ontology

Concept map for notes/viktor/synthetic-agentic-benchmarks/. Built 2026-07-13.

Survey scaffold for the whole space: [[llm-agent-eval-benchmarking-survey]] (taxonomy of eval objectives, interaction modes, difficulty levers, and named benchmarks — read it first to place any single note).

The core problem

Two failure modes recur when you auto-generate agentic benchmarks:

• Lumped difficulty. Generate-a-task-and-hope pipelines emit whatever the base LLM finds easy, so tasks clump in one narrow band. You get no monotonic spread from trivial → frontier, so the set can't separate weak agents from strong ones.
• Artificial quality gates. When correctness can't be computed from the world, teams bolt on an LLM-judge or a brittle rule-check. Those gates are gameable, drift, and reject valid solution paths — they feel forced because they are not grounded in execution.

The fix is two decoupled disciplines: (1) make difficulty a structural knob you can dial and measure, not a prompt adjective; (2) make verification grounded by construction — the answer key exists (and provably executes) before the task ships, so a judge is a last resort, not the primary signal.

Taxonomy of generation strategies

(a) Instruction / data-synthesis lineage (roots of everything; instruction-tuning heritage, transfers to agentic tasks): Self-Instruct → [[evol-instruct-wizardlm-depth-breadth-difficulty]] (rewrite instructions harder along depth+breadth ops, filter failed evolutions) → [[magpie-instruction-synthesis-from-scratch]] (self-synthesis from the empty chat template, then tag-then-filter to shape the distribution) → [[agentinstruct-generative-teaching-agentic-flows]] (agentic flows: transform seeds → taxonomy-driven instructions → Suggester–Editor difficulty escalation).

(b) Forward task synthesis with verifiable-by-construction answers (build the task around a known, checkable answer): [[taskcraft-automated-agentic-task-generation]] (atomic source-grounded task, then depth/width extension), [[logigen-deterministic-state-verification]] (compile policy into DB constraints; forward-search real solution paths; verify by exact state equivalence), [[apigen-mt-tool-trajectory-synthesis]] (blueprint with ground-truth action sequence, validated by executability before any trajectory is collected).

(c) Reverse / backward synthesis (act first in the real world, derive the task from what happened, so the trace is the label): [[os-genesis-reverse-task-synthesis]] (explore GUI, derive atomic→high-level tasks from (state, action, state) triples), [[firefly-backward-task-synthesis-real-apis]] (graph-guided DAG exploration of real APIs, write the task backward from observed outcomes).

(d) Environment synthesis + agent–environment co-evolution (generate/mutate the world, let difficulty emerge and self-correct toward the frontier): [[agent-world-environment-synthesis]] (synthesize thousands of environments + a self-evolving arena that targets the agent's failure gaps), [[genenv-difficulty-aligned-task-coevolution]] (environment simulator as a learned curriculum policy targeting a solve-rate band), [[programmable-graph-evolution-agent-benchmark-difficulty]] (typed relational graph of data/tools/schema; add/remove/modify operators propagate coherently), [[agentsynth-scalable-computer-use-tasks]] (compose easy, individually-verifiable subtasks into hard chains).

How to control the COMPLEXITY DISTRIBUTION (the anti-lumping section)

This is the heart of the user's pain. The collected sources converge on a small set of concrete mechanisms for producing a wide, dial-able spread instead of a lump. Ordered from simplest to most rigorous:

1. Diversity at generation time (upstream anti-lumping). Before any difficulty knob bites, the generator itself collapses to a few typical outputs, so re-sampling one prompt yields near-duplicates. [[verbalized-sampling-mode-collapse]] traces this to typicality bias baked into RLHF preference data — it survives temperature tweaks because it is a data-level cause. The training-free fix: prompt the generator for N candidates with their probabilities ("generate 5 tasks and their probabilities, including unusual/low-probability variants") so it surfaces tail modes instead of the single most-typical one — 1.6–2.1× diversity with no accuracy loss. Fix single-mode generation first; then shape difficulty with the mechanisms below.
2. Structural depth / width extension. [[taskcraft-automated-agentic-task-generation]]: chain atomic tasks (depth = hop count) and fuse independent facts (width = source/tool count). Difficulty = hops × sources, a structural product, not a subjective knob. [[evol-instruct-wizardlm-depth-breadth-difficulty]] supplies the op menu (add-constraint, deepen, increase-reasoning-steps ≈ more action hops) with a bounded per-round increment so difficulty accumulates gradually rather than saturating — then mix tasks from all rounds to get the full easy→hard spread.
3. Subtask-composition levels with measured spread. [[agentsynth-scalable-computer-use-tasks]]: information asymmetry — each subtask is easy to make and verify, but the composite is hard. N subtasks = tier N, and the tiers demonstrably separate agents (18% success at level 1 → 4% at level 6, a ~4.5× drop). This is the cleanest continuous difficulty axis.
4. Step-count / action-count tiers. [[gaia-general-ai-assistants-benchmark]]: tier by the number of steps/tools a competent human needs (L1 ≤5 steps → L3 arbitrarily long). Compositional depth gives an interpretable axis with a monotonic solve-rate gradient across tiers.
5. Human time-to-complete as a continuous axis. [[metr-time-horizon-task-difficulty]]: label each task with the wall-clock time a domain expert needs, and report a model as a time horizon — the human-task-time at which it crosses 50% success. Unlike categorical tiers this axis is continuous, monotonic, and model-independent (anchored to human effort, so it doesn't drift as models improve), so tasks can't lump the way discrete labels do; sample log-uniformly across the time range and the spectrum fills evenly by construction. Frontier time horizon has doubled ~every 7 months.
6. Progressive enhancement until the frontier agent fails. [[progressive-difficulty-synthetic-web-agent-tasks]]: don't set difficulty a priori — the baseline agent is the difficulty meter. Keep amplifying (constraints / more entities / merge subtasks / more hops) until the strong agent can no longer solve it; that failure point marks the retained difficulty. Spread the set by mixing tasks captured at different stopping depths.
7. Solve-rate-band targeting (offline α-curriculum + online per-step filtering). [[genenv-difficulty-aligned-task-coevolution]]: reward the generator with a bell curve R = exp(−β(p̂−α)²) peaked at target solve rate α≈0.5; discard batches outside a ±0.1 band. As the agent improves, mastered tasks lose reward and the generator emits harder variants — an automatically-maintained distribution. Its RL-training counterpart is [[online-difficulty-filtering-rl]]: label difficulty by pass rate over G=16 rollouts against the current policy and keep only prompts in a narrow intermediate band — (0.3, 0.7) empirically wins. The theory is sharp: at p=0 or p=1 the GRPO advantage variance collapses to zero, so always-solved and never-solved prompts carry no gradient signal for training and no discriminative signal for benchmarking; the informative mass is the center band. Filtering is online (recomputed every step) with async replacement to hold batch size — a moving band that tracks the model, i.e. genenv's re-centering done per-step and cheaper. [[agent-world-environment-synthesis]] does the same gap-driven targeting via a self-evolving arena; [[programmable-graph-evolution-agent-benchmark-difficulty]] provides the structural mutation operators to realize the harder variants coherently. The unifying theory behind all of items 6–7 is frontier-targeting (see the auto-spread-difficulty section below): online-difficulty-filtering, GenEnv's α-band, and progressive-enhancement are all discretized instances of maximizing a signal that is zero at both tails — regret ([[paired-unsupervised-environment-design]], [[accel-regret-evolving-curricula]]) or solver-uncertainty-at-0.5 ([[r-zero-challenger-solver]]). Because the signal re-centers as ability grows, difficulty auto-spreads and stays centered on the frontier instead of lumping — this is the principled cure, and the pass-rate band is just its cheap approximation.
8. Psychometric / IRT calibration + audit layer (measure and repair the distribution the above produce):
   • Predict + generate at a target difficulty: [[amortized-irt-difficulty-prediction-llm-eval]] — Rasch-calibrate a seed, train an embedding→difficulty predictor for free labels, then a difficulty-conditioned generator (PPO reward = negative distance to z_target) to fill under-represented bands (0.96 correlation with true calibration).
   • Audit for the lumping failure directly: [[irt-benchmark-audit-saturation-difficulty-ceiling]] (PSN-IRT) — fit difficulty/discrimination/guessing/feasibility, then drop saturated, ceiling, low-discrimination, and low-feasibility items; stratify survivors across bands, prefer high-discrimination within each band. This is the tool that detects "too lumped together."
   • Measure any agent with a tiny adaptive subset: [[fluid-benchmarking-irt-adaptive-llm-eval]] (2PL + MAP + Fisher-info selection) and [[atlas-adaptive-testing-llm-evaluation]] (3PL + EAP + SE stopping + randomesque exposure). Both pick items whose difficulty ≈ the agent's ability (max information), and both let bad items fall out by low discrimination — a non-artificial quality gate driven by fit statistics, not opinion.

Actionable recipe (anti-lumping): first defeat single-mode generation — prompt for N candidates with probabilities ([[verbalized-sampling-mode-collapse]]) so the raw pool is diverse before you shape it. Define difficulty structurally (hops × sources, or subtask count) so it's monotonic and dial-able, and attach a human-time coordinate ([[metr-time-horizon-task-difficulty]]) sampled log-uniformly for an even spread; push each seed up with bounded enhancement ops until a frontier agent fails; keep tasks captured at every stopping depth. Filter/label by pass rate against the current solver and target the (0.3, 0.7) band ([[online-difficulty-filtering-rl]]), dropping always-solved/never-solved items as zero-signal; then IRT-calibrate the pool, drop saturated/ceiling/low-discrimination items, and use a difficulty-conditioned generator to backfill empty bands. Re-measure as agents improve (co-evolution) so the band stays centered on the frontier.

How to build NON-ARTIFICIAL quality gates (grounded verification)

Grounded gate = correctness computed from real state or real execution; artificial gate = a judge/rubric guessing whether output is right. Prefer grounded everywhere it exists.

Executable / state-based checks (read the world's final state, tolerate many solution paths, catch side effects):

• [[webarena-programmatic-functional-correctness-evaluators]] — programmatic reward reads DB row / API / JS state; LLM matching confined to string-format normalization only.
• [[appworld-controllable-apps-benchmark]] — state-based unit tests plus negative assertions that penalize collateral damage (corrupting unrelated data fails the task).
• [[osworld-real-computer-env-benchmark]] — each task bundles (setup config, instruction, execution-based checker script) that inspects post-task OS state.
• [[tau-bench-tool-agent-user-interaction]] — grade final DB state vs annotated goal state; adds an LLM user simulator for conversational friction and a pass^k reliability axis (consistency, not just capability).
• [[tau2-bench-verified-database-state-verification]] — DB-state oracle, human-corrected so the expected state is policy-compliant and its IDs resolve (grounding and calibration are separate jobs).

Execution-cascade verification (cheap→expensive filters, execution is the decisive gate):

• [[apigen-xlam-verifiable-function-calling-pipeline]] — 3-stage cascade: format parse → actually execute the API → semantic check; the load-bearing gate is real execution, judgment confined to intent-alignment after a successful run.
• [[covert-verifiable-tool-use-synthesis]] — build an immutable oracle trajectory first, add difficulty only via oracle-preserving transforms (distractors/noise) so reference-match reward stays valid; judge used narrowly only for behaviors reference-matching can't express.

Human-screened gates (fix the oracle, not the agent's output):

• [[swe-bench-verified-human-quality-gate]] — keep the oracle executable (unit tests) but human-audit it for false negatives and task solvability; triple-annotated 0–3 rubric drops flawed items. The 16%→33% GPT-4o jump was the bad tests' false-negative rate.
• [[tau2-bench-verified-database-state-verification]] and [[swe-bench-verified-human-quality-gate]] share the move: the ground truth was the problem, humans repaired it; the success signal stays executable.
• [[gaia-general-ai-assistants-benchmark]] uses two-annotator validation to guarantee one unambiguous quasi-exact-match answer (no judge at all).

When forced into an LLM judge (last resort, no state/reference check exists):

• [[llm-judge-failure-modes-evidence-grounding]] — failure modes are rubric-execution drift, unverifiable attribution, human-scale misalignment. Mitigate with the Rulers pattern: lock the rubric into a fixed checklist, require extractive quote evidence per decision, and post-hoc calibrate to human score boundaries. Never ship a bare "rate 1–10" gate.

Grounded vs artificial, contrasted: a grounded gate (state diff, execution, reference match) is reproducible and un-gameable — the agent can't talk its way to a pass because only doing the task moves the state. An artificial gate (bare rubric/judge) drifts, can't be audited, and rejects valid alternative paths — exactly the "forced" feeling. The backward/forward-verifiable synthesis strategies (clusters b & c above) are attractive precisely because they make the answer key exist before the task, so the gate is grounded automatically.

Failure modes & mitigations (why benchmarks come out lumped / saturated / gameable)

The critiques cluster names why auto-generated benchmarks fail and, for each, the concrete repair. Read it as the diagnostic index behind the two disciplines above.

Lumped / low-diversity (the user's core pain). Three root-cause fixes, in generation order:

• Generator collapses to typical outputs. [[verbalized-sampling-mode-collapse]]: RLHF typicality bias makes the generator emit near-duplicates; prompt for N candidates with probabilities to surface tail modes (1.6–2.1× diversity, training-free). This is the earliest lever — a diverse raw pool before any difficulty shaping.
• No principled difficulty coordinate. [[metr-time-horizon-task-difficulty]]: human time-to-complete is a continuous, model-independent axis; log-uniform sampling across it fills the spectrum instead of clustering.
• Difficulty labelled statically instead of against the current solver. [[online-difficulty-filtering-rl]]: score by pass rate over G=16 rollouts, keep the (0.3, 0.7) band; p∈{0,1} items are provably zero-signal. This is the concrete "keep tasks at informative difficulty" recipe.

Saturated / contaminated (benchmarks decay and stop discriminating).

• Memorization inflates scores. [[livecodebench-living-benchmark]]: date-stamp every task and evaluate a model only on tasks published after its cutoff — contamination-free by construction; keep the set rolling, not frozen. Contamination is detectable as a score drop across the cutoff line.
• Full mitigation taxonomy. [[contamination-static-to-dynamic-survey]]: protect-static (canaries, encryption, curation, post-hoc detection) vs the durable fix — dynamic/regenerated benchmarks with temporal-cutoff protocols. Treat any fixed synthetic set as perishable; build a regeneration pipeline.
• Corrupted ground truth caps measurable capability. [[are-we-done-with-mmlu]]: a 6.49% label-error rate (57% in some subjects) means a saturated benchmark measures noise; audit a sample, report per-split error rate, keep a small human-re-annotated gold subset (MMLU-Redux pattern). [[irt-benchmark-audit-saturation-difficulty-ceiling]] is the psychometric detector for the saturation/ceiling symptom.

Gameable gates / artificial-feeling verification (the user's second pain).

• Executable verifiers get gamed more as tasks grow. [[specbench-reward-hacking-agents]]: the visible-vs-held-out gap widens ~28pp per 10× code size; a single executable check is insufficient at long horizons. Fix: hold out integration tests the agent never sees, and track the visible-vs-holdout gap as a first-class reward-hacking metric.
• Task authoring ≠ prompt writing. [[good-terminal-agent-task-design-guideline]]: >15% of popular terminal-agent tasks are reward-hackable; a good task is adversarial (catches failure), difficult from conceptual not clerical/environmental complexity, and legible (transparent pass criteria). Audit generated tasks against its six failure modes (AI-authored specs, over-prescriptive specs, clerical difficulty, weak oracle solutions, tests validating wrong behavior, hackable environments). Legibility is what makes non-artificial verification possible.
• When you must judge, judge auditably. [[llm-judge-failure-modes-evidence-grounding]] remains the last-resort pattern (locked checklist + extractive evidence + calibration).

Validity / quality (is the benchmark even measuring the claimed thing).

• [[construct-validity-llm-benchmarks]]: a 445-benchmark review — most lack construct validity (undefined constructs, unrepresentative tasks, no statistical rigor); its 8 recommendations say define the construct first and check the generated distribution actually spans it, the antidote to "lumped tasks measuring one narrow sub-skill under a broad label."
• [[betterbench-benchmark-quality]]: a 46-practice lifecycle checklist (design → implementation → documentation → maintenance) — generation is only step one; a benchmark isn't done without statistical reporting, reproducibility, and a maintenance plan. Use the checklist as a grounded (checkable) quality gate, not an artificial one.

Production frameworks & the RL-env / verifier layer

The production cluster answers "what to actually use or build," concentrating on the RL-environment and verifier plumbing that makes grounded verification operational at scale.

Env-as-package-with-verifier (the organizing pattern). [[prime-intellect-environments-hub]]: a community registry where each environment ships its own reward/verification logic (via the Verifiers library's taskset/harness/runtime split) and publishes per-model evaluation reports. The takeaway is architectural — the verifier travels with the task, so there is no global artificial gate and difficulty/discrimination is visible per environment.

Procedural environment generation (escape hand-authoring).

• [[endless-terminals-scaling-rl-envs]]: a no-human 4-stage pipeline (task description → containerized env → completion test → solvability filter) yielding 3,255 terminal tasks. The per-task completion test is the verifier and the solvability filter is the quality gate — bad tasks are dropped, not gated artificially.
• [[r2e-gym-procedural-envs-hybrid-verifiers]]: SWEGEN mines executable SWE envs from commit history (change → synth tests → back-translated problem), 8.7K+ tasks; verification is a hybrid ensemble of an execution verifier (grounded but low-discrimination) and an LLM judge (high-discrimination but biased) — a direct template for combining organic ground truth with a re-ranking judge.
• [[swe-gym-agents-and-verifiers]]: 2,438 real Python repos bundled with runtime + unit tests + issue spec; the repo tests are the grounded verifier, and it additionally trains trajectory verifiers that score how the agent solved for best-of-N selection without hand-written rubrics.

The scorer contract. [[inspect-ai-aisi-eval-framework]]: the dataset / solver / scorer decomposition is the clean contract — keep the grader (scorer) separate and swappable from the agent (solver) and tasks (dataset); prefer deterministic scorers (includes, custom checkers) as the grounded gate and layer model_graded_qa only where free-form judgment is unavoidable, with a standardized Docker/K8s sandbox for reproducible grading.

The production complexity-control loop. [[online-difficulty-filtering-rl]] is what closes the loop in practice: within the RL/eval harness, difficulty is not a static tag but the live pass rate over G rollouts against the current policy, filtered to the (0.3, 0.7) band per step — the operational mechanism that keeps a generated pool from lumping as the agent trains.

The theory of auto-spread difficulty: regret / learning-progress / frontier-targeting

This section names the principle underneath every anti-lumping mechanism above: score candidate tasks by a signal that is ~zero at both the trivial and the impossible tails, and generation self-concentrates in the solvable-but-unmastered band — which re-centers automatically as the agent improves. This is the RL-theory answer to the user's "too lumped" pain: you never set difficulty by hand, you optimize a frontier signal and the distribution falls out, and it stays centered because the signal moves with ability.

The signal has three interchangeable faces, all of them single-peaked at the frontier:

• Regret (protagonist-vs-antagonist gap): [[paired-unsupervised-environment-design]] founds Unsupervised Environment Design (UED) — an adversary picks environment parameters that maximize regret (how much better an optimal policy would do than the current agent). Regret is near-zero on trivial levels (agent already optimal) and near-zero on impossible levels (nobody can do them), so the maximizer is forced onto the frontier of just-out-of-reach tasks, and the whole set is minimax-regret robust.
• Learnability / learning-progress: a level is worth generating only if the agent can still improve on it — again zero at both tails.
• Solver uncertainty / pass-rate-near-0.5: identical geometry, expressed as "keep tasks the current solver gets right about half the time" (see the self-play cluster below and [[online-difficulty-filtering-rl]]).

Why this cures lumping specifically: a lump is what you get when the generator optimizes typicality (emit whatever is easy/likely). A frontier signal instead optimizes marginal informativeness; mastered tasks lose their score and drop out, so the generator is continuously pushed to vacate the crowded easy band and colonize the next unmastered one. The band is a moving target, not a fixed label, so the pool cannot collapse to one difficulty as long as the loop runs.

The four UED mechanisms, simplest to most compounding:

• [[paired-unsupervised-environment-design]] — the foundational minimax-regret formulation (protagonist vs antagonist vs adversary/generator); establishes why regret-maximization targets the frontier and yields zero-shot transfer.
• [[robust-plr-replay-guided-ued]] — Dual Curriculum Design (DCD) unifies PAIRED's generator with Prioritized Level Replay's curator; Robust PLR (PLR⊥) uses a gradient-stop so the agent trains only on replayed high-regret levels, giving a provable Nash/minimax-regret guarantee at equilibrium. The practical, stable way to run regret-targeting.
• [[accel-regret-evolving-curricula]] — ACCEL evolves the curriculum: mutate/edit the current high-regret levels to breed slightly harder offspring, so complexity compounds into an emergent difficulty ladder that starts trivial and climbs on its own (Adversarially Compounding Complexity by Editing Levels). This is the mechanism that manufactures the spread, not just the frontier point.
• [[omni-epic-interestingness-coded-environments]] — replaces the hand-coded regret proxy with a foundation-model judge of "learnable AND interesting" and generates the environment and its reward function as executable code (Environments Programmed In Code). Shows the frontier signal generalizes to open-ended, code-defined agentic tasks where you can't enumerate a level space — the bridge from grid-world UED to LLM-agent task synthesis.

Practically: [[online-difficulty-filtering-rl]] is the cheap discretized version of exactly this theory (pass-rate band ≈ regret band), and [[genenv-difficulty-aligned-task-coevolution]] / [[agent-world-environment-synthesis]] are its LLM-agent instantiations. Read this section as the why; read the complexity-distribution section as the how.

Self-play & model-written task generation (proposer at the frontier)

The self-play cluster puts the frontier signal inside a proposer model that is rewarded for generating tasks at the solver's edge — the same principle as UED, but the "environment designer" is now an LLM and the frontier is measured on the co-evolving solver. All variants below reduce to "reward the proposer when the solver is neither certain nor helpless":

• [[asymmetric-self-play-alice-bob]] — foundational: Alice proposes goals by acting in the world (goals are realizable by construction), Bob must reproduce them; Alice is rewarded for goals Bob can't yet solve but that are still achievable. The original "proposer paid to sit at the solver's frontier," and goals are grounded (physically reachable) not hallucinated.
• [[r-zero-challenger-solver]] — two copies of one base LLM from zero data: the Challenger's reward is 1 − 2|p̂ − 1/2|, which peaks exactly at a 50% solve rate — an explicit uncertainty reward that is the pass-rate-near-0.5 face of the regret signal. Difficulty auto-spreads because the Challenger is mechanically pushed to the solver's coin-flip band and re-centers as the Solver learns.
• [[absolute-zero-reasoner]] — a single model is both proposer and solver; the proposer's reward is 1 − mean_solve (learnability), and a code executor is the grounded verifier/environment — so tasks are correct-by-execution, not judged. The non-artificial-gate answer and the frontier-reward answer in one loop.
• [[self-challenging-code-as-task]] — the agent writes tasks in a Code-as-Task formalism: an instruction plus a verification function plus test cases. The generated verifier doubles as the RL reward, so the proposer cannot generate a task it can't ground — grounding is structural, and it targets multi-turn tool/agent tasks (not just single-answer reasoning).
• [[spice-corpus-self-play]] — the Challenger mines a large document corpus to synthesize tasks, so the proposer is anchored to external ground truth; SPICE argues this is what prevents the hallucination/collapse of ungrounded self-play (where the proposer drifts into unverifiable or degenerate tasks). Corpus grounding = the diversity-and-validity fix for pure self-play.
• [[anthropic-model-written-evals]] — the earliest at-scale demonstration that a model can author 154 eval datasets via generate-then-model-filter (then crowd-verify), cheaply out-covering hand-written benchmarks and surfacing inverse-scaling behaviors curated sets missed. Evidence that model-written breadth beats human curation on both scale and coverage.

The through-line: regret (UED), uncertainty-at-0.5 (R-Zero), and 1 − mean_solve (Absolute Zero) are the same single-peaked frontier signal wearing different math; whichever you pick, the difficulty distribution becomes an emergent property of the loop instead of a hand-set knob. And the durable answer to "artificial gates" here is grounding the proposer — in a code executor ([[absolute-zero-reasoner]], [[self-challenging-code-as-task]]), a physically-realizable goal ([[asymmetric-self-play-alice-bob]]), or a real corpus ([[spice-corpus-self-play]]) — so the verification is computed, not opined, and the proposer can't reward-hack itself with ungrounded tasks.

Autoresearch for benchmark authoring

Karpathy's canonical Autoresearch is a constrained hill-climbing loop, not a benchmark generator: establish a baseline, change one mutable program, run it under a fixed five-minute budget, keep an improvement, revert a regression or crash, append every outcome to a ledger, and continue until interrupted (experiment loop, agent contract). The comparison is only meaningful because data loading, validation data, metric code, seeds, and budget accounting stay in a fixed harness while architecture, optimizer, and training logic stay in the mutable file (fixed harness). A session can then be published as a self-contained research record: final diff, successful commit chain, rejected/crashed trials, hardware, budget, and outcome summary (PR 44 session record).

Benchmark-authoring inversion. Freeze the benchmark world, scorer, protected reference data, split registry, seeds, budget accounting, and promotion rules; make only task definitions or generator logic mutable. The closest agent-eval adaptation locks a LangSmith dataset and evaluators while mutating agent code; benchmark authoring reverses that surface by fixing the solver and mutating cases or their generator (Autoresearch for Agents). Trace2Evolve adds the missing promotion discipline: generated hard cases steer discovery, while distinct development, frozen holdout, full-regression, safety, and final gates decide acceptance, with split role and provenance stored explicitly (Trace2Evolve).

Reusable scaffolding. Helix turns the contract into helix.yaml: editable/read-only paths, metric direction, evaluation command, timeout, parser, and an optional secondary quality guard; its limitation is that protection is largely contractual unless the runtime isolates read-only assets (Helix). DSBench shows the suite-scale form: one shared framework generates task-local configs, checkpoints, ledgers, reasoning records, winners, and reports, while a held-back test set is touched only by final reporting; it also shows why executable invariants must backstop templates and documentation that can drift (DSBench Autoresearch).

Mechanical containment and stopping. Scalar Loop hash-pins evaluator files, tests, fixtures, and manifests; restricts staging to declared mutable paths; requires both metric improvement and a guard; and hard-bounds subprocesses and iterations. It still supports only one metric plus one guard, has no protected promotion phase, and does not count repeated tamper or verification failures toward plateau stopping (Scalar Loop). Autoresearch Lab moves the mutable pipeline into Docker with read-only evaluator and data mounts while the host owns Git, and adds iteration, elapsed-time, discard, restart, target, and estimated-cost caps. But the agent can repeatedly inspect evaluation data and write its own keep verdict, which the host does not independently recompute; file immutability is not promotion independence (Autoresearch Lab). The benchmark-authoring contract therefore needs sealed paths, bounded resources, and a host-side promotion service that reruns protected evidence outside the agent's writable channel.

Versioned benchmark construction and expansion. Evo gives benchmark creation its own pre-baseline phase: author programmatic or fuzzy cases, emit per-task traces, split visible development cases from held-out exit-code gates, and run a benchmark-reviewer audit before optimization; its held-out slice remains visible, so externally controlled final evaluation is still required for strong claims (Evo Discover). Claude Deep Evolve makes evaluator replacement a privileged Tier 3 outer-loop operation after plateau and coverage evidence: generate the replacement completely, rerun and repeat its baseline, record both harness identities, and treat activation as a benchmark-version boundary rather than comparing raw scores across versions (Claude Deep Evolve). Together they support a two-timescale design: freeze scoring inside each search epoch, then allow audited suite expansion with rebasing between epochs.

Trace mining and self-expanding suites. Claude Code Nightly converts longitudinal user sessions into provenance-rich replay tasks and deterministically stratifies a dated personal suite, but it has no frozen promotion split and its bootstrap resamples stored responses rather than repeated model runs (Claude Code Nightly). Autoresearch Playwright demonstrates the dangerous naive form: a crawler proposes browser coverage while the mutable test suite is optimized on pass ratio, allowing hard tests to be removed and the score to improve without broader capability coverage (Autoresearch Playwright). The authoring loop therefore needs an immutable behavior manifest, case-count and non-duplication floors, independent state assertions, and a separate privileged promotion path from mined development cases into protected regression and final suites.

Objective alignment and experiment isolation. Cerebras reports an Autoresearch compression run that optimized a plausible proxy—masking experts already in memory—without delivering the intended memory reduction; the side investigation then polluted shared context and steered later proposals off-objective. Strict final gates prevented promotion but not wasted compute. Run each candidate in a clean workspace, carry forward only the accepted incumbent plus a compact external ledger, check construct alignment cheaply before expensive solver or judge runs, and budget rejected proposals explicitly (Autoresearch cheating and isolation).

Search method is a controlled variable. Under the same fixed compute, classical TPE/SMAC/CMA-ES beat pure LLM search inside a bounded 14-parameter space; free-form code editing narrowed the gap, and a hybrid LLM+CMA-ES controller did best in the reported setting (Autoresearch vs classical HPO). A benchmark factory should therefore benchmark its authoring controller against random and classical search over the same bounded mutation schema, and report free-form semantic mutation as a separate condition with invalid-candidate rate, seed variance, and total end-to-end cost.

Search-process meta-optimization. Bilevel Autoresearch adds slower outer loops that change search constraints and synthesize mechanisms such as tabu search or bandit allocation while the inner loop edits training code. Its reported 5× mean gain comes from one benchmark with three repeats per condition, high variance, and import validation rather than a semantic improvement gate. For benchmark authoring, treat proposal policy, mutation operators, failure taxonomy, and exploration schedule as versioned outer-loop artifacts; retain the old policy and promote a replacement only when matched inner runs improve accepted-task yield, diversity, leakage, and held-out ranking stability (Bilevel Autoresearch).

Beyond a single scalar. AutoCircuit first applies hard feasibility floors, then retains every nondominated bandwidth–power point instead of hiding trade-offs in a weighted sum; its exact comparisons lack noise tolerances and repeated measurements, so stochastic benchmark scores need confidence-aware Pareto updates (AutoCircuit). Autoreason handles subjective refinement through blind three-way tournaments among the incumbent, an adversarial revision, and a synthesis: shuffled neutral labels, multiple Borda-ranking judges, incumbent-favoring ties, and repeated incumbent wins as convergence. For evaluator evolution, keep the incumbent eligible to win and preserve the full tournament lineage, but require a valid-judge quorum and rank candidates against locked human-scored, executable, adversarial, and hidden cases rather than letting an LLM panel become ground truth (Autoreason).

Statistical hazard: the loop optimizes the measurement. Repeated keep/discard decisions turn the exposed validation set into development data; even canonical Autoresearch findings reversed across sessions or seeds, and validation spoilage was raised but not measured (reproducibility and validation spoilage). More generally, a noisy selection criterion can keep improving after true generalization declines, so the entire authoring procedure—search budget, mutation surface, and keep rule—must be evaluated inside an outer split rather than judging only its selected artifact (model-selection criterion overfitting). If a promotion holdout must be reused, mediate access: ordinary candidates receive coarse feedback, revealing a noisy holdout score spends a finite exposure budget, item-level traces stay hidden, and a fresh final set remains untouched (reusable holdout).

Acceptance needs protected evidence. Prompt optimization on 30 repeatedly inspected examples reached a reported 100%, but that fixed set became development data and the repository did not retain the per-run artifacts needed to replay every score; expose rich case diagnostics only on development data and promote on hidden and adversarial sets (prompt optimization). Claude Autoresearch's core loop still accepts any one-run positive delta despite its comparison document advertising medians and confirmations; its separate regression protocol shows the safer pattern—baseline-to-candidate hard transitions, repeated performance samples, effect thresholds beyond a noise band, and explicit unavailable evidence—but the repository does not mechanically collect every gate. Benchmark promotion should compute these verdicts in protected code from raw repeated measurements and route accepted high-impact changes through an independently isolated verification hop (Claude Autoresearch).

Practical contract. Use discovery traces to propose candidates; compare them on a fixed development suite across multiple task, solver, and evaluator seeds; require non-regression on a technically protected promotion holdout and full suite; expose the final split once after the search and stopping rule are frozen. Keep an append-only ledger of parent candidate, mutation, split exposures, seeds, scores, invalidity/crash reason, and artifact paths. A candidate should optimize a declared multi-axis benchmark objective—construct coverage and discrimination subject to verifier-reliability, safety, leakage, and cost guards—rather than a single gameable score.

Named systems reference table

System	How tasks are authored / generated	Difficulty knob	Verification method
[[evol-instruct-wizardlm-depth-breadth-difficulty]]	LLM rewrites seed instructions harder (5 depth + 1 breadth ops)	Bounded per-round increment over M rounds; mix rounds for spread	Elimination-Evolving filter (drops no-gain / refused / leaked evolutions)
[[magpie-instruction-synthesis-from-scratch]]	Self-synthesis from empty pre-query template	Difficulty tags + reward + neighbor-distance; tag-then-filter	Reward model + quality/difficulty tagging (SFT data, not agentic)
[[agentinstruct-generative-teaching-agentic-flows]]	Agentic flows transform raw seeds → taxonomy-driven instructions	Suggester–Editor refinement loop escalates complexity	Taxonomy coverage + refinement (SFT data pipeline)
[[taskcraft-automated-agentic-task-generation]]	Atomic source-grounded task, then extended	Depth (hop count) × width (source/tool count)	Answer must be locatable in source; generator must solve it first
[[logigen-deterministic-state-verification]]	Architect/Set-Designer/Explorer; forward search for real paths	Task complexity across 8 domains	Exact DB state equivalence to oracle (path-agnostic)
[[apigen-mt-tool-trajectory-synthesis]]	Validated blueprint with ground-truth actions → simulated interplay	Blueprint config (action count/dependency, policy constraints)	Format → executability → LLM-committee; keep only execution-verified
[[apigen-xlam-verifiable-function-calling-pipeline]]	Generate function calls against 3,673 real APIs	API category + call arity (single/parallel/multiple)	3-stage cascade: format → execute → semantic
[[covert-verifiable-tool-use-synthesis]]	Base trajectory + oracle-preserving augmentations	Augmentation params (distractors, ambiguity, noise)	Reference match to preserved oracle; judge only for error-handling
[[os-genesis-reverse-task-synthesis]]	Explore GUI first, reverse-derive tasks from interaction triples	Exploration breadth; atomic→high-level composition	Trajectory Reward Model (graded 1–5: completion + coherence)
[[firefly-backward-task-synthesis-real-apis]]	Sample sub-DAGs over a real tool graph, execute, write task backward	Sub-DAG size/shape (chain length, width)	Intrinsic — real execution trace is the label
[[agent-world-environment-synthesis]]	Synthesize thousands of environments + tasks; self-evolving arena	Gap-driven dynamic synthesis at the failure frontier	Programmatic checks against executable tool ecosystems
[[genenv-difficulty-aligned-task-coevolution]]	Environment simulator as learned curriculum policy	α-curriculum reward peaked at target solve rate (±0.1 band)	Empirical solve-rate measurement (sample agent k times)
[[programmable-graph-evolution-agent-benchmark-difficulty]]	Typed relational graph; add/remove/modify operators	Transformation depth/breadth (nodes touched, propagation)	Coherent propagation keeps env internally consistent
[[agentsynth-scalable-computer-use-tasks]]	Compose easy, independently-verifiable subtasks into chains	Number of component subtasks (6 tiers)	Each subtask cheaply checkable; composite inherits checks
[[gaia-general-ai-assistants-benchmark]]	Human-authored, anchored to stable sources	3 tiers by required steps/tools	Quasi-exact-match, type-normalized (no judge)
[[webarena-programmatic-functional-correctness-evaluators]]	241 templates → 812 intents across 4 self-hosted sites	Multi-step navigation + config depth	Programmatic reward reads DB/API/JS state; fuzzy_match for format only
[[appworld-controllable-apps-benchmark]]	Curated multi-app tasks over pre-populated user state	Normal vs challenge tiers; breadth of apps/APIs coordinated	State-based unit tests + collateral-damage negative assertions
[[osworld-real-computer-env-benchmark]]	(setup config, instruction, checker) triple per task	Cross-app multi-step workflows	Per-task execution-based checker inspects OS state
[[tau-bench-tool-agent-user-interaction]]	Policy + tools + tasks; LLM-simulated user	Conversational friction; pass^k reliability axis	Final DB state vs annotated goal state
[[tau2-bench-verified-database-state-verification]]	τ²-bench tasks, human-corrected ground truth	4 domains	DB final-state comparison; expected state audited for policy/IDs
[[swe-bench-verified-human-quality-gate]]	Real GitHub issues + repo unit tests, human-screened	Time-estimate labels (<15min … >4hr)	Executable FAIL_TO_PASS tests; human-audited oracle
[[prime-intellect-environments-hub]]	Community registry; each env submitted with its own verifier	Per-environment (no global knob); per-model eval reports	Verifier travels with the task (Verifiers taskset/harness/runtime split)
[[endless-terminals-scaling-rl-envs]]	4-stage no-human pipeline: task desc → container → completion test → solvability filter (3,255 tasks)	Task-type diversity + scale (no explicit knob)	Per-task completion test in container; solvability filter as gate
[[r2e-gym-procedural-envs-hybrid-verifiers]]	SWEGEN mines commits → synth tests → back-translated problem (8.7K+ tasks)	Organic from real commit size/complexity; scale	Hybrid ensemble: execution verifier (grounded) + LLM judge (discriminative)
[[swe-gym-agents-and-verifiers]]	2,438 real Python repos + runtime + unit tests + issue spec	Real-repo variance	Repo unit tests (executable) + trained trajectory verifiers for best-of-N
[[inspect-ai-aisi-eval-framework]]	Framework: dataset + solver + scorer contract, @task registration	(framework, not a generator)	Deterministic scorers first (includes/custom); model_graded_qa only when unavoidable

The synthesized SOTA recipe

An opinionated end-to-end pipeline drawing the best idea from each cluster. Goal: a well-spread, auto-verifiable synthetic agentic benchmark with no artificial gates.

1. Ground the world first. Stand up a real executable environment (APIs/tools/apps) with rich pre-populated multi-user state, à la [[appworld-controllable-apps-benchmark]] / [[osworld-real-computer-env-benchmark]]. Compile domain policies into DB constraints so "followed policy?" reduces to "is the end state legal?" ([[logigen-deterministic-state-verification]]).
2. Prefer backward synthesis for free labels. Let a strong agent explore the tools first and derive tasks from what actually happened — sample sub-DAGs over a tool graph ([[firefly-backward-task-synthesis-real-apis]]) or reverse-derive from interaction triples ([[os-genesis-reverse-task-synthesis]]). The execution trace is the answer key, so verification is intrinsic. Where you must go forward, generate a blueprint with a ground-truth action sequence and validate it by executability before collecting any trajectory ([[apigen-mt-tool-trajectory-synthesis]]); seed atomic tasks whose answer is locatable in a source ([[taskcraft-automated-agentic-task-generation]]).
3. Make difficulty structural and dial-able — from a diverse pool. First beat mode collapse: prompt the generator for N candidates with probabilities so the raw pool spans the space instead of clustering ([[verbalized-sampling-mode-collapse]]). Then compose easy, individually-verifiable subtasks into chains (tier = subtask count, [[agentsynth-scalable-computer-use-tasks]]); extend by depth×width ([[taskcraft-automated-agentic-task-generation]]); use the depth/breadth op menu from [[evol-instruct-wizardlm-depth-breadth-difficulty]] / [[agentinstruct-generative-teaching-agentic-flows]] with bounded increments. Tag every task with a step/tool count ([[gaia-general-ai-assistants-benchmark]]) and a human-time coordinate sampled log-uniformly ([[metr-time-horizon-task-difficulty]]).
4. Calibrate difficulty against a live agent. Push each seed up until a frontier agent fails ([[progressive-difficulty-synthetic-web-agent-tasks]]), and drive generation with an α-curriculum reward targeting a ~0.5 solve-rate band that re-centers as agents improve ([[genenv-difficulty-aligned-task-coevolution]], [[agent-world-environment-synthesis]]). In the loop, label difficulty by pass rate over G=16 rollouts against the current solver and keep the (0.3, 0.7) band — always-solved/never-solved items are zero-signal ([[online-difficulty-filtering-rl]]). Retain tasks captured at every stopping depth to keep the full spread. Better still, don't filter after generation — generate from the frontier: drive a proposer/curriculum that scores candidate tasks by a frontier signal (regret, or solver pass-rate near 0.5) so difficulty auto-spreads by construction and re-centers as the agent improves — the minimax-regret UED formulation ([[paired-unsupervised-environment-design]]), its stable replay-guided form ([[robust-plr-replay-guided-ued]]), its evolutionary compounding-difficulty form ([[accel-regret-evolving-curricula]]), a foundation-model "learnable + interesting" judge over code-defined tasks ([[omni-epic-interestingness-coded-environments]]), or an LLM proposer rewarded at the solver's edge — 1−2|p̂−½| ([[r-zero-challenger-solver]]) or 1−mean_solve ([[absolute-zero-reasoner]], [[asymmetric-self-play-alice-bob]]). Ground the proposer so it can't drift into unverifiable tasks: anchor it to a code executor ([[absolute-zero-reasoner]]), a self-written verification function ([[self-challenging-code-as-task]]), or a real document corpus ([[spice-corpus-self-play]]) — ungrounded self-play collapses. And note model-written generation scales cheaply past hand curation ([[anthropic-model-written-evals]]).
5. Verify grounded, never artificial — verifier travels with the task. Bundle each task with its own reward/checker rather than a global gate ([[prime-intellect-environments-hub]]), and keep the grader swappable from the solver via a dataset/solver/scorer contract ([[inspect-ai-aisi-eval-framework]]). Grade by final-state diff / execution, not transcript ([[webarena-programmatic-functional-correctness-evaluators]], [[tau-bench-tool-agent-user-interaction]]). Cascade cheap→expensive with execution as the decisive gate ([[apigen-xlam-verifiable-function-calling-pipeline]]); keep an immutable oracle and add hardness only via oracle-preserving transforms ([[covert-verifiable-tool-use-synthesis]]); add negative assertions for collateral damage ([[appworld-controllable-apps-benchmark]]). At long horizons hold out integration tests and track the visible-vs-holdout gap to catch reward hacking ([[specbench-reward-hacking-agents]]), and author tasks to be adversarial/legible so passing implies solving ([[good-terminal-agent-task-design-guideline]]). Where a judge helps only for re-ranking, ensemble it with the execution verifier rather than replacing it ([[r2e-gym-procedural-envs-hybrid-verifiers]], [[swe-gym-agents-and-verifiers]]); use a bare LLM judge only as a last resort, in the locked-checklist + extractive-evidence + calibration form ([[llm-judge-failure-modes-evidence-grounding]]).
6. Audit the oracle, not just the agent. Human-screen a sample of tasks/tests for false negatives, solvability, and policy-consistent expected state ([[swe-bench-verified-human-quality-gate]], [[tau2-bench-verified-database-state-verification]]). Grounding and calibration are separate jobs — do both.
7. IRT-calibrate and prune the final pool. Fit item difficulty/discrimination/feasibility, drop saturated / ceiling / low-discrimination / low-feasibility items, stratify survivors across bands ([[irt-benchmark-audit-saturation-difficulty-ceiling]]). Backfill empty bands with a difficulty-conditioned generator ([[amortized-irt-difficulty-prediction-llm-eval]]).
8. Evaluate adaptively + report reliability. Score any new agent with a tiny Fisher-information-selected subset targeted at its ability ([[fluid-benchmarking-irt-adaptive-llm-eval]], [[atlas-adaptive-testing-llm-evaluation]]), and report pass^k consistency, not just pass@1 ([[tau-bench-tool-agent-user-interaction]]).
9. Keep it living, not frozen. Treat any fixed set as perishable: date-stamp every task and only score a model on tasks postdating its cutoff ([[livecodebench-living-benchmark]]), and run a regeneration/refresh cadence with temporal-cutoff protocols so memorization never pays off ([[contamination-static-to-dynamic-survey]]).
10. Gate on quality, not just correctness. Before trusting scores, define the construct and check the pool actually spans it ([[construct-validity-llm-benchmarks]]); run the lifecycle checklist (report significance, reproducibility, maintenance plan) as a grounded gate ([[betterbench-benchmark-quality]]); and human-re-annotate a gold subset to measure the label-error rate, treating >~5% as capability-obscuring ([[are-we-done-with-mmlu]]).

Open questions / gaps

• No end-to-end system does all three well. Backward-synthesis methods ([[firefly-backward-task-synthesis-real-apis]], [[os-genesis-reverse-task-synthesis]]) give grounded labels but don't IRT-calibrate; the IRT audit tools ([[irt-benchmark-audit-saturation-difficulty-ceiling]], [[atlas-adaptive-testing-llm-evaluation]]) assume a response matrix over many models, which is expensive for a fresh agentic benchmark. Stitching cluster (c/d) generation to cluster (f) calibration is unproven at scale.
• IRT for agentic tasks is under-validated. Nearly all IRT evidence here is on static QA/MCQ benchmarks (MMLU, GSM8K, HellaSwag). Whether item difficulty/discrimination behave the same for multi-turn, stateful, tool-using tasks — where "an item" spans a whole trajectory — is not established.
• Cost of the co-evolution loop. Solve-rate-band targeting ([[genenv-difficulty-aligned-task-coevolution]]) requires sampling the agent k times per candidate task every epoch; the sources don't quantify the compute overhead of maintaining the distribution vs. a static generate-once set.
• Reliability (pass^k) as a difficulty dimension is barely explored. Only [[tau-bench-tool-agent-user-interaction]] treats consistency as first-class; none of the generators optimize for producing tasks that separate agents on reliability rather than one-shot capability.
• Collateral-damage / side-effect checking is rare. Only [[appworld-controllable-apps-benchmark]] verifies what must stay unchanged. Most grounded verifiers assert the goal state but not the absence of damage, leaving a blind spot for destructive-but-goal-completing agents.
• Contamination of the generator itself. Sources address benchmark contamination of the agent under test, but not what happens when the task-generating LLM has memorized the source corpora it synthesizes from.
• No validated selection rule for autonomous benchmark authoring. Karpathy's loop has one stable metric, while AutoCircuit preserves explicit Pareto trade-offs and Autoreason supplies a blind tournament for subjective artifacts; neither establishes which benchmark-quality vector, confidence rule, or human/executable reference set yields externally valid agent rankings (Autoresearch experiment loop, BetterBench, AutoCircuit, Autoreason).
• Reusable-holdout guarantees do not directly cover agentic benchmark tasks. Thresholdout assumes bounded statistical queries over IID samples, while stateful tool trajectories, generated task families, and shared environments introduce dependencies whose sensitivity is not characterized (reusable holdout).
• Free-form semantic search lacks a fair benchmark-authoring baseline. Classical optimizers win in bounded Autoresearch parameter spaces, while Bilevel Autoresearch reports outer-loop gains on one training benchmark; there is no matched comparison for open-ended task/verifier edits where mutation validity, judge cost, and search-space definition differ (Autoresearch vs classical HPO, Bilevel Autoresearch).
• Noise-aware promotion is specified but not end-to-end enforced. Claude Autoresearch documents repeated sampling, effect thresholds, holdout routing, and unavailable-evidence states, but its core loop still accepts one-run positive deltas and its reducer trusts preclassified inputs; no collected system mechanically isolates the holdout and collects every regression dimension (Claude Autoresearch).
• No collected harness closes the full promotion trust boundary. Scalar Loop seals files and edit scope but lacks hidden promotion and cost budgets; Autoresearch Lab isolates evaluator files but exposes evaluation data and trusts an agent-writable verdict. No source combines tamper-proof scope, hidden evidence, independent reruns, functional GPU/package access, and secret-safe network controls (Scalar Loop, Autoresearch Lab).
• Tiered evaluator expansion lacks external validity evidence. Claude Deep Evolve implements versioned harness replacement and Evo audits constructed gates, but neither source publishes a long-running corpus showing that autonomous suite expansion improves construct coverage or external discrimination rather than only internal test coverage (Claude Deep Evolve, Evo Discover).
• Trace-mined and crawled tasks still need a grounded admission criterion. Session duration, tool use, and UI discovery identify substantial candidate flows, not necessarily reproducible or discriminating tasks; the sources do not establish an autonomous gate that converts these signals into valid protected benchmark cases (Claude Code Nightly, Autoresearch Playwright).

Coverage summary (built 2026-07-13)

Total: 53 notes across 8 collection angles — (1) landscape/taxonomy of generation strategies, (2) complexity-distribution control, (3) grounded (non-artificial) verification, (4) named systems deep-dives, (5) critiques / contamination / saturation, (6) production RL-env & verifier frameworks, (7) automatic curriculum / Unsupervised Environment Design (UED), (8) self-play & model-written task generation.

The 3-line answer to the user's pain:

1. Un-lump difficulty by targeting a frontier signal at GENERATION time. Score/propose tasks by solver pass-rate ≈0.5 / regret / learnability (single-peaked at the frontier, zero at both tails), so the distribution self-concentrates in the solvable-but-unmastered band and re-centers as ability grows ([[paired-unsupervised-environment-design]], [[accel-regret-evolving-curricula]], [[r-zero-challenger-solver]], [[genenv-difficulty-aligned-task-coevolution]]). Add verbalized-sampling ([[verbalized-sampling-mode-collapse]]) for tail-mode diversity and online (0.3, 0.7) pass-rate filtering ([[online-difficulty-filtering-rl]]) as the cheap discretized version.
2. For non-artificial gates, prefer grounded executable/state verifiers over LLM-judge. Compute correctness from a code executor, DB/OS state diff, unit tests, or corpus grounding ([[absolute-zero-reasoner]], [[self-challenging-code-as-task]], [[webarena-programmatic-functional-correctness-evaluators]], [[appworld-controllable-apps-benchmark]], [[spice-corpus-self-play]]); reserve rubric-grounded LLM judges for the genuinely unverifiable residue, in the locked-checklist + extractive-evidence + calibration form ([[llm-judge-failure-modes-evidence-grounding]]).
3. Refresh with living/rolling benchmarks to defeat contamination. Date-stamp tasks and score only post-cutoff, run a regeneration cadence with temporal-cutoff protocols ([[livecodebench-living-benchmark]], [[contamination-static-to-dynamic-survey]]).

Open gaps the collected sources don't answer well:

• Cross-domain difficulty calibration for OPEN-ENDED agentic tasks. Frontier signals are well-defined per-environment, but no source shows how to make "difficulty 0.6" mean the same thing across heterogeneous open-ended domains (web vs SWE vs GUI vs reasoning) so a single pool spreads coherently.
• Measuring the QUALITY of the difficulty distribution itself. Everything optimizes toward the frontier band, but there is no accepted metric for whether a finished pool has a good spread (coverage, monotonicity, discrimination across bands) beyond post-hoc IRT — no target-distribution objective.
• Multimodal / GUI task synthesis at scale. Grounded verification is easiest for code/DB/API; visual-grounded GUI tasks ([[osworld-real-computer-env-benchmark]], [[os-genesis-reverse-task-synthesis]]) are represented but the sources don't show frontier-targeted generation working at scale for pixel/vision-grounded agents.
• Cost/compute of frontier-targeted generation loops. Every proposer/co-evolution method requires repeatedly sampling the solver k times per candidate each epoch; none quantify the compute overhead vs a static generate-once set, nor when the loop stops paying for itself.
• Grounding the proposer against its own memorization. SPICE flags ungrounded self-play collapse, but no source addresses task-generator contamination — a proposer that has memorized its grounding corpus/executor test suite emits leaked or degenerate "frontier" tasks.

Autoresearch benchmark-authoring extension (built 2026-07-20)

Added: 24 newly integrated notes across 8 angles — (1) Karpathy's canonical loop and protected harness, (2) agent/eval adaptations, (3) suite construction and privileged expansion, (4) validation spoilage and adaptive holdouts, (5) classical, hybrid, and bilevel search, (6) scope sealing, isolation, resource caps, and promotion boundaries, (7) Pareto and blind-tournament selection, and (8) trace-mined and self-expanding suites.

Practical synthesis:

1. Freeze the construct, scorer, schemas, protected cases, solver budget, split registry, and promotion rules; expose only task definitions or generator logic to the inner authoring loop (fixed harness, Trace2Evolve).
2. Hash-seal protected files, allowlist mutable paths, run each candidate in a clean workspace, and keep Git plus final keep/revert authority outside the agent. A container is useful process isolation, not proof that promotion evidence is independent (Scalar Loop, Autoresearch Lab, Autoresearch cheating and isolation).
3. Give the agent rich per-case diagnostics only on development data. Select across repeated task, solver, and evaluator seeds with a measured noise floor; promote through hidden holdout, regression, safety, leakage, and final gates whose raw evidence is recomputed by protected code (prompt optimization, Claude Autoresearch, reusable holdout).
4. Apply non-negotiable validity floors first, then preserve nondominated alternatives across discrimination, coverage, novelty, verifier agreement, cost, ambiguity, and leakage. For subjective evaluator changes, use blinded incumbent-versus-challenger tournaments only as one signal against locked references (AutoCircuit, Autoreason).
5. Bound iterations, wall time, per-command time, rejected-candidate spend, cumulative restarts, and no-promotion plateaus independently. Record every parent, mutation, environment hash, split exposure, seed, raw measurement, decision, crash, and cost in an append-only ledger (agent contract, Autoresearch Lab).
6. Keep scoring fixed inside a search epoch. Treat generator, mutation-policy, or evaluator changes as privileged outer-loop version transitions, rebaseline them, and compare old versus new policies over matched inner runs before activation (Claude Deep Evolve, Bilevel Autoresearch).

Remaining Autoresearch gaps:

• No published end-to-end benchmark factory combines sealed scope, hidden independent promotion, noise-aware multi-objective selection, versioned outer-loop changes, and complete cost accounting.
• No study validates which Pareto objectives or subjective-tournament protocol best preserve external agent rankings across benchmark versions.
• Reusable-holdout guarantees remain unclear for dependent, stateful trajectories and generated task families rather than bounded IID queries.
• Practical isolation remains unresolved when evaluators require GPUs, package caches, emulators, or network services without exposing secrets or protected data.
• Proposal precision and stopping are not compute-optimal: sources cap spend after the fact, but do not predict which candidate is worth an expensive multi-agent evaluation or when further search ceases to justify its cost.