The scope increase was not simply the predictable result of “more tests.” The new iree-test-suites/LLM corpus and systematic bug injection invalidated several assumptions under which the earlier ConSan prototype appeared usable. Small kernels had spare registers, simple code-object ownership, familiar LDS barriers, and faults that could be represented by dropping one instruction. Production IREE models instead supplied register-saturated kernels, multiple processes and code objects, mixed FLAT/VGLOBAL/LDS operations, subtle atomic protocols, and enough runtime complexity that a run could look successful even when no intended kernel had been instrumented. Fault injection then made semantic incompleteness observable: an engine could run cleanly yet miss a wrong address, weakened ordering, failed compare-exchange, or slightly displaced barrier. The resulting work is the machinery needed to make the original claim—usable end-to-end concurrency sanitization—true and demonstrable on that corpus.
-
The first broad matrices exposed vacuous success. Instrumentation sometimes applied to a helper, loader, or unrelated code object rather than the intended dispatch; in other cases a requested engine produced no records. We therefore had to add owner-qualified discovery, exact code-object/kernel identities, application-count checks, source/output oracles, and retained manifests. This is why much of the diff is campaign and provenance infrastructure rather than detector logic.
-
Real model kernels exhausted the prototype’s scratch-register assumption. The MOI engines needed temporary VGPRs/SGPRs but optimized LLM kernels often had no safe dead window. That made spilling, descriptor/resource updates, code relocation, state preservation, and fail-closed handling prerequisites rather than optimizations. Kunwar’s relocation/spilling branch provided an important starting point, but it did not by itself cover ConSan’s varied insertion sites, automatic resource planning, special-register preservation, or the qualification needed for arbitrary production binaries.
-
Malformed and adversarial inputs exposed unsafe patching behavior. SuperCollider could hang or leave partially transformed objects when presented with exactly the kinds of broken synchronization patterns users want a sanitizer to diagnose. Hardening required typed transform outcomes, transactional mutation, final ELF/resource validation, bounded execution, quarantine, and fuzz/adversarial corpora. These changes are not extra product features; they prevent the diagnostic tool from becoming a second failure source.
-
Subtle injections showed that “atomic support” was not one feature. Wrong-address, wrong-scope, weakened order/fence, RMW/bitwise, and compare-exchange faults require distinct facts: exact effective address, memory role, scope, operation class, and dynamic CAS outcome. The existing representation could not express all of them. That led to shared semantic discovery plus engine-specific work: success-predicated CAS publication, acquire-before-release replay, ordinary load/store association, explicit unsupported classifications, and trace-ABI evolution.
-
Each scalable MOI flavor failed differently. Record/replay needed bounded, complete causal windows rather than an unstructured event list. Sampled mode revealed torn/stale snapshot and generation-consistency false positives. InlineShadow needed sound address partitioning and wave-level coalescing so that moderate overhead did not discard divergent-address correctness. These are separate algorithms sharing discovery and spilling infrastructure; one small common patch could not make all three faithful.
-
The LLM corpus made qualification itself a deliverable. IREE end-to-end tests span subprocesses, cached artifacts, large model assets, and long GPU runs. Reproducibility required immutable input locks, exact manifests, health gates that explicitly remove instrumentation from
rocminfo/smoke probes, serialized GPU access, resumable matrices, and JSON/CSV evidence. One recent “GPU failure” was in fact a health probe contaminated by ConSan’sREQUIRE_RECORDSsetting—an example of why this infrastructure cannot be hand-waved away.
Measured from the parent of the first PLAN_INJECTION.md commit through the current committed HEAD, the change is about 52,282 added and 2,305 removed lines across 100 files. Approximately 35,848 additions (69%) are tests/harnesses, 4,645 (9%) are documentation/plans, and 11,789 (23%) are production or build code. The raw size therefore overstates the sanitizer implementation churn: most additions encode fault fixtures, non-vacuity checks, provenance, retained evidence, and regressions for failure modes discovered by the campaign.
The coherent PR-level purpose is: make ConSan safely applicable to optimized production GPU binaries and prove, with injected concurrency defects, what each engine catches without corrupting clean workloads. Spilling enables application; semantic modeling enables detection; hardening makes broken inputs safe; and retained end-to-end matrices prove both claims. Removing any one of those layers would restore the earlier possibility of a green but vacuous or unsound result.
That does not mean every current line is irreducible. Before review, we should split the history into understandable vertical changes, consolidate duplicated runners/fixtures, keep generated evidence out of the source diff where possible, and present per-engine acceptance tables. The justification for the large body of work is strong, but the final review surface should still be smaller and more structured than the development history that discovered it.