CES run 2026091483495627 resolved 74 of 89 Terminal-Bench 2.1
instances (83.1%). This is a nominal absolute-quality improvement of two
resolves over historical pure Opus 5 run 2026091315708394, which recorded
72/89 (80.9%). A second historical pure Opus 5 full arm recorded 71/89
(79.8%).
This is a service-level CES endpoint comparison, not a new contemporaneous
control experiment. The treatment's engine-artifact coverage was incomplete:
87 of 89 completed CES outputs contained awf-report.json. The two missing
engine reports (qemu-alpine-ssh and qemu-startup) failed during generic Git
bootstrap before AWF started. This does not change CES's reported 74/89
endpoint, but it limits per-instance workflow-trace analysis for those two
tasks.
| Measure | Treatment | Historical pure Opus 5 |
|---|---|---|
| CES resolved | 74/89 | 72/89 |
| CES resolve rate | 83.1% | 80.9% |
| Absolute difference | +2 resolves | baseline |
| Complete AWF reports | 87/89 | 86/89 |
The treatment preserves a working first attempt instead of asking several agents to make unrelated patches and then trying to select among them. Gemini 3.8 Flash first changes the task checkout directly. Claude Opus 4.8 receives a separate retained-source worktree containing that Gemini state and may make one focused repair. GPT-5.5 compares the Gemini incumbent and Opus repair using task- and source-grounded evidence. An explicit tie keeps the Gemini incumbent.
flowchart TD
A[Original task checkout at HEAD] --> B[Gemini 3.8 Flash: implement initial solution]
B -->|initial completed| C[Freeze retained Gemini incumbent]
B -->|initial unavailable| H[Return initial result; no repair]
C --> D[Claude Opus 4.8: one isolated retained-source repair worktree]
D --> E[Collect incumbent and repair diffs]
E --> F[GPT-5.5: source-grounded pairwise review]
F -->|Opus is concretely better| G[Publish selected Opus repair]
F -->|Gemini wins or explicit tie| I[Keep Gemini incumbent]
F -->|malformed or conflicting reviews| J[One signal-recovery review]
J -->|clear winner| G
J -->|still unresolved| I
The topology is deliberately bounded:
| Role | Model | Workspace | Responsibility |
|---|---|---|---|
| Incumbent author | Gemini 3.8 Flash | Original task checkout | Produce the first complete task-relevant implementation. |
| Repair challenger | Claude Opus 4.8 | Isolated retained-source worktree | Inspect the incumbent, identify a concrete omission or regression, and make one focused repair only when justified. |
| Selector | GPT-5.5 | Disposable read-only judge lease | Compare the two source states against the complete task and return a structured winner or tie. |
Full task and diff evidence never travel in the judge command-line argument.
The engine writes a lease-local .awf-pairwise-evidence.md file inside the
GPT judge's disposable workspace. GPT receives only a short fixed instruction
to read that file and return exactly one structured verdict:
{"winner":"A"}Valid alternatives are {"winner":"B"} and {"winner":"tie"}. The parser
also retains compatibility with older bare A, B, tie, and winner: ...
responses, but the treatment requests JSON.
sequenceDiagram
participant E as AWF executor
participant J as GPT-5.5 judge lease
E->>J: Create scratch workspace
E->>J: Write .awf-pairwise-evidence.md
Note right of J: Complete task<br/>selection context<br/>incumbent diff<br/>repair diff
E->>J: Fixed prompt: read evidence file and return JSON
J-->>E: {"winner":"A" | "B" | "tie"}
E->>E: Parse structured verdict
alt A/B selected
E->>E: Preserve the selected source state
else Both directional reviews explicitly tie
E->>E: Retain Gemini incumbent without recovery
else Reviews disagree or are unreadable
E->>J: One recovery comparison
end
The selector is a source-reading model judgment, not a substitute for the external Terminal-Bench verifier. It is not given evaluator assets, hidden tests, grading metadata, gold patches, prior task outcomes, or network access. Candidate instructions also prohibit installation/downloads, check weakening, and manufactured success claims.
retain_source: true reserves candidate index zero for the completed Gemini
state. With best_of: 2, Opus supplies the sole challenger at index one. This
means the baseline work survives an inconclusive repair/review rather than
being overwritten by a speculative second attempt.
When a challenger is selected, AWF first preflights and applies its indexed patch directly to the original checkout. A three-way patch application is used only if actual source drift prevents direct application. This makes normal publication exact while retaining an explicit source-drift recovery path.
| Setting | Value |
|---|---|
| Benchmark | terminalbench2-1 |
| CES run | 2026091483495627 |
| Cohort | Explicit full 89-instance list |
| Cohort SHA-256 | 3146396679086dabf17348787d160c7bac5f31adbee442d6a89577cb6a052bb1 |
| Flow | tb21-gemini-opus-gpt-repair |
| Flow SHA-256 | 8e2e08198ed562290906a7da7349736548837af242849a384bc7433ee50906d6 |
| Backend | worktree |
| Candidate topology | Retained Gemini incumbent plus one sequential Opus challenger |
| Pairwise selector | GPT-5.5, file-backed structured verdict |
| CES workers | 30 |
| Workflow and candidate timeout | 1,800,000 ms |
| Declared budget | aiu=1200 |
| Publication mode | uncommitted |
| Engine source recorded at build | feat/persistent-flow-memory@4345002+dirty |
Among the 77 instances with recorded candidate detail, 69 were decided by an explicit tie. Those ties retained the Gemini incumbent; 65 of the 69 tie-path instances resolved and four failed. This is an observation about the executed arm, not evidence that ties themselves cause resolution or that the judge proved the selected implementation correct.
The canonical engine implementation is committed as
microsoft/agent-workflow-engine@d993b50 on
feat/persistent-flow-memory. The CES runtime staging, full cohort, timeout
forwarding, and engine-report coverage guard are committed as
github/copilot-agent-runtime@f58371c1cd on
huamin/workflow-engine-integration.
The detailed study record, including historical comparisons and the
engine-artifact limitation, is maintained in docs/HANDOFF.md.