| Field | Value |
|---|---|
| Copilot CLI Run | 25953625783 |
| Claude Code Run | 25953626217 |
| Model | claude-sonnet-4.6 (both runs) |
| Benchmark | SWE-Bench Verified (500 tasks) |
| Copilot CLI Result | 371/500 (74.3%) |
| Claude Code Result | 358/500 (71.9%) |
| Delta | +2.4% (Copilot CLI ahead) |
| Category | Count | Pass Rate Impact | Description |
|---|---|---|---|
| Both Pass | 339 | -- | No action needed |
| Both Fail | 109 | -- | Model-limited (same model, both fail) |
| Copilot CLI Only | 30 | -- | Copilot CLI advantage -- protect these |
| Claude Code Only | 19 | +3.8% potential | Improvement targets (agent scaffolding gap) |
| Env Noise | 3 of 19 | +0.6% | Network/infra flaky (psf__requests x2, matplotlib-25311) |
| Actionable | 16 of 19 | +3.2% potential | Real scaffolding fixes available |
| Fix Pattern | Tasks Addressable | Pass Rate Delta | Effort |
|---|---|---|---|
| Exact-target verification gate | 8 | +1.6% | Low -- prompt text |
| Edge-case completeness review | 7 | +1.4% | Low -- prompt text |
| Benchmark-aware test strategy | 6 | +1.2% | Low -- prompt text |
| Problem grounding before coding | 3 | +0.6% | Low -- prompt text |
Note: Tasks overlap across patterns. Fixing all 16 actionable tasks would raise Copilot CLI from 74.3% to ~77.5% (+3.2pp). The top two fixes (verification + edge-case) alone cover 12 of 16 tasks.
Data extracted from
custom_metrics.jsonin each task's output zip. Claude Code lacks per-request latency/token metrics in its output format.
| Metric | Copilot CLI | Claude Code | Delta |
|---|---|---|---|
| Time P50 (sec) | 115.0 | 64.5 | +78% slower |
| Time P90 (sec) | 430.6 | 250.7 | +72% slower |
| TTFT P50 (sec) | 1.6 | 2.2 | 37% faster |
| Cost P50 (USD) | $0.28 | $0.28 | -- |
| Cost P90 (USD) | $1.06 | $0.92 | +15% costlier |
| Request Latency P90 (ms) | 17,683 | N/A | -- |
| Tokens Total P90 | 52,557 | N/A | -- |
| Cost Total (USD) | $228.08 | $232.28 | 1.8% cheaper |
| Cache Write Tokens (avg) | 27,571 | N/A | -- |
| Tool Calls (avg) | 27.7 | N/A | -- |
| Compaction Count (avg) | 0.0 | N/A | -- |
| Metric | Copilot CLI | Claude Code | Delta |
|---|---|---|---|
| Time P50 (sec) | 128.9 | 91.3 | +41% slower |
| Time P90 (sec) | 356.4 | 454.2 | 22% faster |
| TTFT P50 (sec) | 1.6 | 2.2 | 37% faster |
| Cost P50 (USD) | $0.30 | $0.40 | 25% cheaper |
| Cost P90 (USD) | $0.89 | $1.30 | 32% cheaper |
| Request Latency P90 (ms) | 20,834 | N/A | -- |
| Tokens Total P90 | 51,854 | N/A | -- |
| Cache Write Tokens (avg) | 25,632 | N/A | -- |
| Tool Calls (avg) | 28.7 | N/A | -- |
- CLI is slower overall: P90 time 430.6s vs 250.7s (+72%) across all 500 tasks. On the 19 failing tasks, the gap narrows and reverses at P90 (356s vs 454s) — Claude Code outlier failures burn more time, but median is still faster (91s vs 129s).
- CLI is costlier overall: P90 cost .10 vs .90 (+22%) across all 500 tasks. On the 19 failing tasks specifically, CLI is cheaper at P90 (.89 vs .30) — suggesting CLI exits earlier on wrong approaches while CC persists longer.
- TTFT advantage: Copilot CLI first-token latency is consistently faster (1.6s vs 2.2s P50) — likely due to prompt caching.
- No compactions: Zero compaction events across all 500 CLI tasks — context window management is working well.
Note: Claude Code output zips don't include per-request latency, token, or cache metrics. Only time, TTFT, and cost are comparable. Full latency/token comparisons would require Kusto telemetry from the
ces_telemetry_prodcluster.
| Pattern | Tasks | Fix |
|---|---|---|
| Incomplete Verification | 8 tasks | Run exact FAIL_TO_PASS test before declaring done |
| Edge-Case Blindness | 7 tasks | Review all return paths, error handlers, type-varying inputs |
| Benchmark-Unaware Tests | 6 tasks | Inspect FAIL_TO_PASS test names; prefer additive tests |
| Wrong Problem Framing | 3 tasks | Reproduce exact failing case before coding |
| # | Task | Category | Hotspot | Root Cause |
|---|---|---|---|---|
| 1 | django-11532 | incomplete_fix | edge_case_miss | Unconditional IDNA encoding broke non-IDNA-safe hostnames |
| 2 | django-11749 | incomplete_fix | test_collision | Helper command arg names collided with evaluator patch |
| 3 | django-11885 | incomplete_fix | incomplete_verification | Wrong expected query count (2 vs 3) |
| 4 | django-12125 | incomplete_fix | test_collision | Test naming mismatched FAIL_TO_PASS |
| 5 | django-12663 | incomplete_fix | incomplete_verification | Fragile LazyObject fix, skipped edge case |
| 6 | django-14559 | incomplete_fix | edge_case_miss | Missed empty-input early-return path |
| 7 | django-16631 | incomplete_fix | incomplete_verification | Secret-fallback session logic mismatch |
| 8 | django-16877 | unknown | incomplete_verification | Structured finding missing |
| 9 | matplotlib-23476 | incomplete_fix | premature_stop | Fixed logic but never added required regression test |
| 10 | matplotlib-25311 | pre_existing | wrong_target | Unrelated PASS_TO_PASS failure (env noise) |
| 11 | seaborn-3069 | incomplete_fix | edge_case_miss | invert_yaxis() corrupted explicit user limits |
| 12 | psf__requests-1724 | correct_approach | (env noise) | Environment/network-dependent test failure |
| 13 | psf__requests-2317 | correct_approach | (env noise) | Environment/network-dependent test failure |
| 14 | xarray-6461 | incorrect_fix | wrong_target | Wrong scalar-attrs fallback strategy |
| 15 | sympy-12096 | incomplete_fix | incomplete_verification | Collateral compatibility/test breakage |
| 16 | sympy-12481 | incomplete_fix | edge_case_miss | Partial Python 3.10 cleanup missed collections.Callable |
| 17 | sympy-15017 | incorrect_test | test_collision | Modified canonical test incompatibly |
| 18 | sympy-15976 | correct_approach | test_collision | Fix worked but test changes misaligned |
| 19 | sympy-17655 | wrong_root_cause | wrong_target | Patched __add__ instead of __rmul__ |
Hotspot: [edge_case_miss] | Root Cause: Unconditional IDNA encoding in get_fqdn_punycode() broke non-IDNA-safe hostnames
Key Decision: Agent used socket.getfqdn().encode('idna') without guarding for non-IDNA domains; later reverted a guard and kept the fragile path.
Full analysis
Hotspot: [test_collision] | Root Cause: Helper command used incompatible arg names (--shop-id/--shop vs expected), blocking evaluator patch
Key Decision: Parser fix was correct; test-command naming drift caused evaluator incompatibility.
Full analysis
Hotspot: [incomplete_verification] | Root Cause: Wrong expected query count (2 vs 3)
Full analysis
Hotspot: [test_collision] | Root Cause: Test naming mismatch -- test_serialize_inner_class_fields vs expected test_serialize_nested_class
Key Decision: Production fix in _serialize_path was correct; test name didn't match FAIL_TO_PASS.
Full analysis
Hotspot: [incomplete_verification] | Root Cause: Fragile LazyObject fix, skipped edge case
Full analysis
Hotspot: [edge_case_miss] | Root Cause: Early return if not objs: return wasn't updated to return 0
Key Decision: Main loop return-count fix added, but empty-input path missed; local tests passed and agent stopped.
Full analysis
Hotspot: [incomplete_verification] | Root Cause: Secret-fallback session get_user() logic didn't match test_get_user_fallback_secret
Key Decision: Fallback key checking added; exact failing test behavior wasn't inspected.
Full analysis
Hotspot: [incomplete_verification] | Root Cause: Structured finding missing
Full analysis
Hotspot: [premature_stop] | Root Cause: _original_dpi overwritten during FigureCanvasBase.__init__ in unpickle on HiDPI
Key Decision: Fix attempted to restore _original_dpi from mutated state; still missing required regression test test_unpickle_with_device_pixel_ratio.
Full analysis
Hotspot: [wrong_target] | Root Cause: Unrelated PASS_TO_PASS failure (env noise)
Full analysis
Hotspot: [edge_case_miss] | Root Cause: Unconditional ax.invert_yaxis() in _finalize_figure corrupted explicit user limits
Key Decision: Agent guarded auto-limit setting with if axis_key not in p._limits but made invert_yaxis() unconditional. invert_yaxis() swaps ylim, so (-1, 2.1) becomes (2.1, -1.0). 20 verification turns never tested the combination.
Full analysis
Hotspot: [wrong_target] | Root Cause: Wrong scalar-attrs fallback strategy (attrs[0] path)
Key Decision: Ternary fallback edit landed; no substantive regression test added.
Full analysis
Hotspot: [incomplete_verification] | Root Cause: Collateral compatibility/test breakage
Full analysis
Hotspot: [edge_case_miss] | Root Cause: Incomplete Python 3.10 cleanup -- matrices.py still used removed collections.Callable
Key Decision: Core fix changed duplicate-cycle handling; remaining compat issues not fully removed.
Full analysis
Hotspot: [test_collision] | Root Cause: Modified canonical test incompatibly
Full analysis
Hotspot: [test_collision] | Root Cause: Fix worked but test changes misaligned with evaluator
Full analysis
Hotspot: [wrong_target] | Root Cause: Misdiagnosed as __add__ issue; actual fix needed in Point.__rmul__
Key Decision: Root-cause identification targeted wrong method.
Full analysis
File: src/prompts/cli/system.ts lines 256-261
Before:
const code_change_instructions = codingChangeInstructions.with({
linting_building_testing: `
* Only run linters, builds and tests that already exist. Do not add new linting, building or testing tools unless necessary for the task.
* Run the repository linters, builds and tests to understand baseline, then after making your changes to ensure you haven't made mistakes.
* Documentation changes do not need to be linted, built or tested unless there are specific tests for documentation.
`,After:
const code_change_instructions = codingChangeInstructions.with({
linting_building_testing: `
* Only run linters, builds and tests that already exist. Do not add new linting, building or testing tools unless necessary for the task.
* Run the repository linters, builds and tests to understand baseline, then after making your changes to ensure you haven't made mistakes.
* When fixing bugs, identify the specific failing test(s) early. Run those targeted tests after each code change -- don't rely only on broad test suite runs which may include unrelated pre-existing failures.
* When your fix involves multiple interacting behaviors (e.g., axis inversion + limit setting), explicitly test the combinations. Edge cases at behavior intersections are where bugs hide.
* If test failures appear unrelated to your changes (e.g., dependency version mismatches), note them as pre-existing and focus verification on the tests directly relevant to your fix.
* Documentation changes do not need to be linted, built or tested unless there are specific tests for documentation.
`,File: src/prompts/cli/system.ts line 225
Before:
- **Verify before claiming success** - Before calling `task_complete`, produce evidence the work satisfies the request: run the relevant tests/build/lint, reproduce the original symptom and confirm it's gone, or otherwise check the result.
After:
- **Verify before claiming success** - Before calling `task_complete`, produce evidence the work satisfies the request: run the relevant tests/build/lint, reproduce the original symptom and confirm it's gone, or otherwise check the result. When your fix touches multiple code paths or behaviors, verify each combination -- bugs often hide at the intersection of interacting features.
File: src/prompts/shared/codingRules.ts line 23
Before:
.with({
validationRules: "* Always validate that your changes don't break existing behavior",
});After:
.with({
validationRules: `* Always validate that your changes don't break existing behavior
* After implementing a fix, verify it handles edge cases -- especially interactions between the changed code paths (e.g., conditional branches that may conflict with each other)`,
});File: src/agents/definitions/rubber-duck.agent.yaml line 67
Before:
- Comments about pre-existing bugs / non-blocking issues in the code which would distract the main agent or lead to scope creepAfter:
- Comments about pre-existing bugs / non-blocking issues in the code which would distract the main agent or lead to scope creep
- When the fix touches multiple code paths that interact (e.g., conditional branches for different features applied to the same data), verify that all combinations are covered -- especially the case where both features are active simultaneously