Thanks for freezing the DAG. That is the reasonable thing to do now. You can take the next 2 hours to fully wrap up. It's 7 AM now; you can take until 9 AM if necessary.
Can you help me understand: What is the shape of the useful new scope that you kept discovering? Did you literally find new bugs in existing code that were only discovered by the new testing scope? Or was it more like pulling threads?
Both happened, but the dominant pattern was: New tests found literal bugs; fixing or validating each bug exposed a deeper missing invariant. Some concrete examples:
-
Real optimized binaries found actual preservation bugs:
- SuperCollider clobbered live v3 in the shared atomicAdd helper.
- Stream-K analysis decoded executable padding as instructions.
- A displaced ds_load_b32 v0, v0 lost its address because the load overwrote the address register before instrumentation captured it.
- Record/Replay could return before its report stores were globally visible.
- Record/Replay instrumentation inserted between a release atomic and the guest’s bookkeeping, creating a race that did not exist in the original program.
-
Fault injection found actual experiment-validity bugs:
- “Nth matching instruction” could mutate the wrong kernel/helper/clone.
- Dropping only half of a split barrier produced an unmatched wait and GPU hang, rather than the intended concurrency fault.
- Campaign collectors initially allowed incomplete or stale rows to look like valid results.
- A passing workload, a ConSan diagnosis, a timeout, and a driver reset were not cleanly distinguished.
-
Once those were repaired, the experiments exposed real detector failures:
- Two substantial atomic campaigns produced zero differential catches.
- Sampled instrumentation collapsed distinct dynamic LDS addresses.
- Inline metadata lacked full dispatch identity, permitting stale cross-dispatch evidence.
- Direct acquired tokens did not provide transitive producer → middle → consumer ordering.
- Current release publication is a plain one-bit validity protocol performed after the guest atomic; it cannot prevent torn release metadata or represent the release-time causal frontier.
That last category is where “pulling threads” became important. For example:
Missed injected atomic bug → inspect acquire/release evidence → discover release is published after the guest → require pre-guest reservation and post-linearization commit → discover metadata and causal snapshot must be one transaction → discover acquire must read the complete transaction with version bracketing → discover inherited tokens need source-release provenance → discover the current token ABI cannot represent that provenance
So some new scope was directly bug fixing; some was the minimum machinery needed to make a fix truthful. There was also scope that was less essential to the product diff: campaign adapters, evidence retention, very detailed DAG bookkeeping, and duplicated test orchestration. That portion should later be consolidated.
My informed breakdown is roughly:
- Preservation and semantic correctness bugs: unquestionably necessary.
- Trustworthy fault targeting and outcome classification: necessary to make any detection claim.
- Causal metadata/protocol work: necessary once the campaigns proved the old detector model insufficient.
- Extensive bespoke harness and planning volume: useful during investigation, but not all irreducible and a likely target for PR consolidation.
The important point is that this was not mostly speculative architecture work. The new corpus repeatedly falsified assumptions that the focused fixtures had allowed us to believe. The “thread pulling” happened because the first-layer fixes were not enough to restore a sound end-to-end