GitNexus A/B — devils-advocate review WITH vs WITHOUT gitnexus, 2 rounds (stale vs fresh index)

gitnexus-ab-summary.md

GitNexus A/B — Devils-Advocate Review

Same Mage-OS plan (14 tasks). Same agent (Opus). Same prompt. Pass A = grep-only. Pass B = WITH gitnexus MCP.

Ran twice — Round 1 against a 105-commit-stale index, Round 2 after re-indexing.

Headline numbers

	Round 1 (stale index)	Round 2 (fresh index)
Cost A → B	$3.61 → $3.82 (+$0.21)	$4.71 → $3.77 (−$0.94)
Time A → B	462s → 377s (−85s)	573s → 338s (−235s)
Critical caught (A / B)	6 / 5	3 / 5
Unique Critical only-A	4	1
Unique Critical only-B	3	3
Pass B variance R1 → R2		5 → 5 Critical (stable)
Pass A variance R1 → R2		6 → 3 Critical (noisy)

Verdict

• Stale index → coin-flip. B costs marginally more, catches a different mix of bugs but misses ones grep finds. "Complementary".
• Fresh index → B wins outright. Cheaper, faster, more thorough, and lower run-to-run variance. Single standout: B caught a 3-stage CLI pipeline that ships structurally broken (empty-string sentinels in the validate stage make every cluster fail import) — grep-only missed it both rounds.

What B catches that A misses (consistently)

1. Cross-method data-flow bugs visible only when you walk the call graph (the CLI-pipeline killer was found via context() on the command class).
2. "Bulk path skips observer" — paper-only mitigation; A couldn't see the negative (scraper doesn't actually set the flag) without enumerating engine callers first.
3. Downstream vendor side effects — orphan catalog_product_link_attribute_int rows after the writer's direct DELETE. Caught via impact() downstream of saveProductLinks.

What A catches that B misses

Historical/documentation drift — the plan said cherry-pick SHA e82c1af55, the real HEAD applied dd64c83f3. Gitnexus doesn't index git history.

Takeaway

Keep gitnexus IF you keep the index fresh. 105 commits stale = bring-your-own-grep. Fresh = clear win.