RDR source-bundle implementation plan

Reconstruction status

This document records the archived prototype/source-bundle target. The clean reconstruction has commits 1 through 10, including the separation of public metadata identity from implementation/link transport. Its historical phase checkmarks do not claim that the reconstructed tree has completed full span ownership, source ownership, or source-bundle work. In particular, Cargo and Buck source-bundle production and staging remain follow-up work.

North star

At every dependency edge, compare the producer's complete observable delta

changed .rmeta;
changed .spans;
changed, added, or removed named source-bundle children;

with that consumer's previously recorded dep-info:

rerun(consumer, producer) iff
    changed_outputs(producer)
        ∩ previously_observed_inputs(consumer, producer)
    is non-empty

A consumer reruns only when an artifact it previously observed changed. If it reruns, its own outgoing artifact edge uses the consumer's newly measured output delta, not the original upstream edit's classification. Independently available upstream providers retain their own deltas for any real transitive artifact edge the later consumer previously observed.

Consequently:

an A .rmeta change may rerun B while unchanged B artifacts keep C cached;
an A .spans change does not rerun B unless B previously observed A's .spans;
an A source-child change does not rerun B unless B previously observed that exact named child;
if C previously observed an A-owned transitive artifact directly, A's delta remains relevant to the real A → C edge regardless of whether B ran;
the same rule applies at every later edge and to every combination of changed artifacts.

This rule is artifact-agnostic. A source edit may change one, two, or all three artifact kinds. Tests and build integrations must measure the artifacts; they must never infer the delta from the edit's intended category.

Every implementation and review agent must receive this section at the start of its prompt. Stop and escalate before accepting any change that:

eagerly reads an artifact merely to transport it;
treats .rmeta as a global rebuild sentinel;
hashes, tags, or fingerprints the source bundle only as one directory;
traverses all source children while staging a transitive bundle;
loses the original owner of a transitive span;
propagates A's edit classification through a rerun of B;
places source text, a whole-file hash, whole-file length, line tables, or normalization tables in .spans;
adds validation, sanitization, repair, authentication, compatibility branches, or fallback lookup inside the trusted rustc/build-system contract;
adds a new source-remapping mechanism before proving that the existing remapping machinery cannot express the required name.

Intended final design

RDR exposes three independently observable tiers:

Consumer need	Inputs that may be recorded
Semantic information	`.rmeta`
Stable source identity, hygiene, selected offsets, or selected line/column anchors	`.rmeta`, `.spans`
Exact source text, layout, character selection, checksum, or embedded source	`.rmeta`, `.spans`, one or more exact named source children

The artifacts have these responsibilities:

.rmeta contains the semantic contract and opaque span references.
.spans contains only the selected positional projection: original owner, stable logical source identity, and the selected normalized endpoint anchors.
<extern-stem>.source-bundle/ is a build-system-owned directory containing the original bytes of files rustc read for that producer, addressed by their existing remapped logical names.
<extern-stem>.source-bundle-names is staging metadata used to construct a transitive symlink tree. It is not a compiler-observable RDR artifact and must not itself become a reason to rerun a consumer compilation.

The source bundle travels beside the producer's .rlib or .rmeta and .spans; it is not stored in the archive and is not a compiler-emitted .sources artifact. Each emitted Rust artifact owns its own sidecars. A crate's bundle contains only that crate's inputs, never copied inputs from a dependency.

The build system compiles through its existing canonical source view and uses the existing rustc path-remapping rules to give every file a stable logical name. The producer wrapper reads rustc's Make dep-info after compilation and copies the current crate's actually-read inputs into the source bundle under those logical names. Declared, mapped, generated, included, and relevant OUT_DIR inputs all use this one path.

The downstream compiler receives source bundles as available context without opening them. Projected operations lazily read .spans. Exact operations first read .spans, then open only the owning crate's named bundle child and record that physical child in dep-info. One read constructs one complete SourceFile in the existing global SourceMap; there is no partial SourceFile and no second BytePos coordinate world.

The physical bundle root is invocation-local context. It never enters .rmeta, .spans, stable hashes, or displayed source identity. Only the existing remapped logical key is serialized; dep-info separately records the physical child that the current invocation opened.

For A → B → C, B may transport the availability of A's artifacts without observing them. If C needs an A-owned position or source child carried through B's metadata, C records A's .spans or source child directly. B must not decode and republish A's position into B-owned slots.

Trust boundary and deliberate non-goals

The contract is established by one controlled producer invocation and consumed by a later controlled compiler invocation. The producer's canonical source view and existing remapping rules establish bundle keys. The consumer trusts the associated bundle.

Do not add compiler-side checks that bundle keys are relative, contained, unique, host-compatible, or content-authenticated. Do not reverse-remap, search the host filesystem, compare a serialized source hash, repair a missing entry, or fall back to an alternate path. A requested child that is absent fails at the ordinary file-open boundary and indicates a compiler/build-graph bug.

Buck's existing infallible failed-action protocol is outside this successful producer/consumer trust boundary. A failed action may materialize an empty declared directory solely so Buck can complete failure filtering, provided the failure status prevents that placeholder from ever entering a RustArtifactFiles provider or consumer action.

Parsing rustc's Make dep-info is a real external-format boundary and should produce an exact, narrow representation once. Repeated downstream validation is not part of the design.

This plan does not:

put source files inside .rlib, .rmeta, or .spans;
make every downstream action read every transitive source;
replace coarse metadata behavior when -Zrdr is disabled;
preserve wire compatibility with prototype .spans files;
add a Cargo integration for producing source bundles. A non-Buck integration that enables RDR exact cross-crate source access must provide the same associated bundle contract.

Confirmed starting point

The current stack already has separate semantic, decode-layout, hygiene-layout, and span-layout identities in compiler/rustc_middle/src/metadata.rs:567. RDR trace, projection, full, reuse, and spans-only encoder states are in compiler/rustc_metadata/src/rmeta/encoder.rs:139, and green .rmeta reuse replays the closed span layout before emitting .spans at compiler/rustc_metadata/src/rmeta/encoder.rs:4397.

Three current behaviors must be corrected:

Compiletest represents a dependency as either Rebuilt or Reused { changed_spans }. A changed .rmeta discards simultaneous .spans information and forces every consumer to run (src/tools/compiletest/src/runtest.rs:1325, src/tools/compiletest/src/runtest.rs:1577, and src/tools/compiletest/src/runtest.rs:3224).
RDR re-encoding calls span.data() for an already-external span, which reads the origin crate's .spans and re-homes the position in an intermediate crate merely to transport it (compiler/rustc_span/src/span_encoding.rs:636, compiler/rustc_span/src/lib.rs:1353, and compiler/rustc_metadata/src/rmeta/encoder.rs:607).
.spans serializes complete SourceFile records, including whole-file hashes, lengths, lines, multibyte data, and normalization positions (compiler/rustc_metadata/src/rmeta/mod.rs:447 and compiler/rustc_span/src/lib.rs:1984).

The existing mechanisms to retain are:

Span's opaque external owner/low-slot/high-slot/context representation in compiler/rustc_span/src/span_encoding.rs:139;
lazy .spans dep-info recording in compiler/rustc_metadata/src/rmeta/decoder.rs:170;
Session::file_depinfo and the existing late dep-info write in compiler/rustc_interface/src/passes.rs:578 and compiler/rustc_driver_impl/src/lib.rs:299;
RealFileName's physical and remapped identities and FilePathMapping::to_real_filename in compiler/rustc_span/src/lib.rs:271 and compiler/rustc_span/src/source_map.rs:1190;
metadata's removal of the local physical path in compiler/rustc_metadata/src/rmeta/encoder.rs:1319;
Buck's existing canonical source construction in ../buck2/prelude/rust/sources.bzl:76;
Buck's current remapping in ../buck2/prelude/rust/build.bzl:500 and ../buck2/prelude/rust/tools/rustc_action.py:349;
Buck's existing RDR artifact tag and Make depfile conversion in ../buck2/prelude/rust/build.bzl:883 and ../buck2/prelude/rust/build.bzl:1685;
Buck directory depfile child selection, covered by ../buck2/tests/core/executor/test_dep_files.py:118.

RustSourcesTSet is not the source-bundle model. It is broad, transitive, unowned, and includes declared rather than actually-read roots (../buck2/prelude/rust/sources.bzl:9). Reusing it would erase both producer ownership and named-child observation.

Execution protocol

The checkboxes in this file and the Codex harness task tracker are both required records. The root agent updates both as work progresses.

For every phase:

The root agent repeats the North Star in the task prompt.
The root agent creates a fresh jj change in the repository that owns the phase. Only the root agent may run jj or create, describe, rebase, squash, or abandon changes. Git is never used.
The root agent spawns a fresh implementation subagent for that phase.
The subagent may inspect and edit only the phase's scope. It may not run any build, test, formatting, snapshot-blessing, Buck, jj, or git command. It reports the commands it wants the root agent to run.
For Rust edits, the subagent follows /rust-style; public Rust doc comments also follow /rustdoc. The root agent applies /stack-review to every phase before presentation.
The root agent reviews the actual diff against this plan and the North Star, then runs all formatting, builds, and tests. A reviewer subagent may be used, and its review may be trusted, but root still owns the phase gate.
A small, local divergence may be decided and recorded by the root agent. Any divergence that changes artifact ownership, observation granularity, remapping, the trust boundary, or phase semantics is escalated to the user.
Only after the root confirms the phase is correct and its required checks pass may the checkbox be marked complete and a dependent phase begin.

No phase crosses the Rust and Buck repositories. Rust and Buck phases that are independent in the dependency graph may be implemented concurrently in their separate repositories, but the root serializes all build/test/format commands and independently reviews each change before releasing either successor.

Dependency graph and ledger

Phase 1: compiletest oracle
├── Rust lane: Phase 2 → Phase 3 → Phase 4 → Phase 5 → Phase 6 ─┐
└── Buck lane: Phase 7 → Phase 8 → Phase 9 ────────────────────┤
                                                               └→ Phase 10

Phase 10 → Phase 11 → Phase 12
Phase 10 → Phase 13
Phase 10 → Phase 15
Phase 12 + Phase 13 → Phase 14
Phase 12 + Phase 15 → Phase 16
Phase 14 + Phase 16 → Phase 17

Phase 1 — Compiletest's artifact-agnostic oracle

Repository: Rust
Depends on: current stack
Purpose: make every later phase directly reviewable before changing the compiler's source representation.

Implementation

Modify:

src/tools/compiletest/src/runtest.rs
src/tools/compiletest/src/runtest/tests.rs
src/tools/compiletest/src/directives.rs
src/tools/compiletest/src/directives/handlers.rs
src/tools/compiletest/src/directives/directive_names.rs
focused fixtures under tests/incremental/rdr/

Replace MetadataReuse with a provider-indexed artifact-delta map that retains exact changed paths for all three artifact kinds at once. Use a narrow artifact identity shape, not independent optional flags and not a Rebuilt catch-all. Aggregating several producers unions paths within their distinct provider entries.

Parse the consumer's prior Make dep-info into exact dependency paths. Remove substring matching. The scheduling operation is exactly:

available_changed_paths = union(provider_deltas.values())
should_run = !available_changed_paths.is_disjoint(prior_dep_info_paths)

Building an auxiliary returns two related facts:

the auxiliary's own freshly measured output delta;
the provider-indexed deltas for every transitive artifact that remains available to its parent.

When deciding whether to run B, flatten the changed paths of B's available providers and intersect them with B's prior dep-info. If B is skipped, B's own entry is empty. If B runs, replace only B's entry with B's freshly measured delta. Preserve A's entry for a genuine A → C transitive artifact edge; C will ignore it naturally when C's prior dep-info did not name an A artifact.

Add these revision-scoped byte-oracle directives:

//@ [bpass2] rdr-rmeta: same
//@ [bpass2] rdr-spans: different
//@ [bpass2] rdr-source: path/to/a.rs different
//@ [bpass2] rdr-source: path/to/unchanged.rs same

same and different describe bytes, not query state and not whether rustc ran. They remain separate from rustc-not-invoked. For a named child, different also covers appearance or disappearance between snapshots.

Add a revision-scoped physical source directive:

//@ [bpass1] revision-source: a_first.rs
//@ [bpass2] revision-source: a_second.rs

Compile the selected physical file while mapping both names to the one logical fixture identity, such as a.rs, through the existing --remap-path-prefix machinery. Do not copy it to a temporary logical filename and do not add a rustc renaming flag.

The harness should stage and snapshot named fixture sources in the same <extern-stem>.source-bundle/ layout used by Phase 10. Phase 1 may model a source-child delta before rustc can open that child downstream, but the harness layout and directives must be final rather than throwaway test infrastructure.

Use the existing built-in rustc_expected_metadata_state crate attribute in the physical revision sources. It asserts the incremental query/work-product state from compiler/rustc_incremental/src/persist/clean.rs:136; the new compiletest directives assert emitted bytes. Neither substitutes for the other.

Tests

Add compiletest unit tests proving:

simultaneous .rmeta, .spans, and two source-child changes all survive aggregation;
exact dep-info intersection handles escaped paths without substring matches;
two sibling source leaves change independently;
source-child additions and removals remain named deltas rather than becoming a directory-wide change;
an unobserved source child does not schedule the consumer;
after B reruns, B's own entry is replaced by B's measured delta while an independently observed A entry remains available to C;
C can observe an A-owned transitive .spans or source child directly without causing B to observe it;
physical revision names map to one logical artifact identity.

Add passing A → B → C incremental fixtures for:

A private positional edit: A .rmeta is the same, .spans is different, the edited source is different, an untouched sibling is the same, and semantic-only B and C are both rustc-not-invoked.
An A semantic edit that is irrelevant to B's exported result: A .rmeta differs and B runs; B's artifacts remain the same, so C is rustc-not-invoked.

Do not add a knowingly failing source-only end-to-end fixture. Unit-test that oracle state now; add the passing fixture in Phase 11 after .spans no longer contains the whole-file source hash.

Root verification

./x fmt --check
./x test src/tools/compiletest
./x test tests/incremental/rdr/<private-position-fixture>
./x test tests/incremental/rdr/<semantic-propagation-stop-fixture>
./x test tests/incremental/rdr

Completion gate

The root review must be able to trace a three-artifact delta through A, B, and C without any global rebuild state or inferred edit category.

Phase 2 — Preserve transitive external-span ownership

Repository: Rust
Depends on: Phase 1
Purpose: make transport of an upstream span observation-free.

Implementation

Modify the existing span encoding path in:

compiler/rustc_span/src/lib.rs
compiler/rustc_span/src/span_encoding.rs
compiler/rustc_metadata/src/rmeta/mod.rs
compiler/rustc_metadata/src/rmeta/encoder.rs
compiler/rustc_metadata/src/rmeta/decoder.rs

Add an RDR wire case that carries the original owning crate plus low and high external slots. Override RDR's SpanEncoder::encode_external_span behavior so it encodes those values directly instead of reconstructing a span and calling span.data().

Local spans still allocate local position slots. Every encoded occurrence still participates in the closed MetadataSpanLayout, including opaque external references, but a pass-through external occurrence does not create an intermediate-owned position entry. Coarse metadata retains its current exact conversion.

Decode the external case back into Span::new_external_with_slots using the original owner. Do not load the owner's .spans during decode or re-encode.

Tests

Add focused rustc_span tests for one-owner/two-slot preservation, including composed endpoints under that owner. Add an A → B → C RDR fixture in which:

B carries an A-owned span but never asks for its position;
B's dep-info does not name A's .spans;
C asks for the position and records A's .spans directly;
B's own .spans does not acquire a copied A position.

Root verification

./x fmt --check
./x test compiler/rustc_span
./x test compiler/rustc_metadata
./x test tests/incremental/rdr/<transitive-owner-fixture>
./x test tests/run-make/rdr-position-outputs
./x test tests/run-make/rdr-hygiene-identity

Completion gate

Review the dep-info and encoded ownership, not just output equality. Any pass-through path that calls Span::data(), lo(), or hi() fails the phase.

Phase 3 — Encode the final projected source/location model

Repository: Rust
Depends on: Phase 2
Purpose: give position-only consumers a durable API that observes .spans without observing source bytes.

Implementation

Modify:

compiler/rustc_span/src/lib.rs
compiler/rustc_span/src/source_map.rs
compiler/rustc_span/src/span_encoding.rs
compiler/rustc_middle/src/metadata.rs
compiler/rustc_metadata/src/rmeta/mod.rs
compiler/rustc_metadata/src/rmeta/encoder.rs
compiler/rustc_metadata/src/rmeta/decoder.rs
compiler/rustc_metadata/src/rmeta/decoder/cstore_impl.rs

Introduce final domain types for:

the producer-exported StableSourceFileId and a source-bundle key derived from the existing fully remapped filename;
a selected normalized endpoint containing its source-relative byte offset, line, byte column, character column, and display column;
a projected span whose endpoints can retain distinct slots and source identities under the one original owner already represented by Span.

The bundle-key type is internal and constructed by the trusted remapping path; it does not validate or sanitize the value.

Change the RDR position table to encode projected identities and endpoint anchors rather than raw producer-global BytePos values. Do not include a source hash, source length, source text, line table, multibyte table, or normalization table in that projected record.

Add a projected-location callback/query for opaque external spans and a SourceMap entry point for consumers that require only identity and selected anchors. This is deliberately distinct from the existing exact EXTERNAL_SPAN_DATA path:

projected access observes .spans;
exact access observes .spans and, after Phase 10, a named source child.

Temporarily retain the existing serialized SourceFile table solely for exact operations. Adapt exact resolution to use the new durable projected endpoints with that table. Do not create a projected SourceFile, a truncated SourceFile, duplicate external files, or another global coordinate space.

Tests

Cover:

ASCII, tabs, multibyte characters, and display columns;
CRLF and BOM normalization;
doctest line offsets and remapped real filenames;
spans with endpoints composed from separate slots under the same owner;
local and external spans returning equivalent projected coordinates;
projected lookup recording .spans but no source file.

Assert the intended artifact rule: changing a selected offset, line, character column, or display column changes .spans. At this phase the retained exact SourceFile table may still cause additional .spans changes; Phase 10 removes it.

Root verification

./x fmt --check
./x test compiler/rustc_span
./x test compiler/rustc_metadata
./x test tests/run-make/rdr-position-outputs
./x test tests/incremental/rdr

Completion gate

The projected representation must be the representation retained in the final design. Reject a temporary offset scheme or a second partially populated SourceFile.

Phase 4 — Migrate general locations, panic locations, and the public bridge

Repository: Rust
Depends on: Phase 3
Purpose: remove exact-source observation from common position-only paths.

Implementation

Route position-only operations through the projected API in:

compiler/rustc_span/src/source_map.rs:432
compiler/rustc_middle/src/mir/consts.rs:519
compiler/rustc_const_eval/src/const_eval/machine.rs:210
compiler/rustc_public_bridge/src/context/impls.rs:345
direct callers exposed by semantic navigation

Use the projected source identity and anchors for filename/location formatting, caller locations, panic locations, const-eval locations, and public bridge line reporting. Preserve local-span behavior through the same public operation.

Do not keep parallel “RDR location” and “ordinary location” call graphs. Deepen the canonical operation so callers choose projected versus exact by the information they request.

Tests

For local and external spans, compare filenames, lines, byte columns, character columns, display columns, panic locations, and bridge output. Dep-info for an external position-only case must contain .rmeta and .spans but no source child.

Root verification

./x fmt --check
./x test compiler/rustc_span
./x test compiler/rustc_const_eval
./x test compiler/rustc_public_bridge
./x test tests/run-make/rdr-position-outputs
./x test tests/incremental/rdr/<projected-location-fixtures>

Completion gate

Semantic navigation must show no remaining Span::lo()/hi() or exact SourceFile request in these position-only paths.

Phase 5 — Migrate coverage to projected locations

Repository: Rust
Depends on: Phase 4
Purpose: keep coverage sensitive to selected coordinates without opening source text.

Implementation

Modify:

compiler/rustc_codegen_llvm/src/coverageinfo/mapgen/covfun.rs:131
compiler/rustc_codegen_llvm/src/coverageinfo/mapgen/spans.rs:38
compiler/rustc_codegen_llvm/src/coverageinfo/mapgen.rs:175
associated coverage tests

Use projected source identity and byte/line anchors for coverage file tables and region coordinates. Keep LLVM coordinate ordering checks on already-shaped projected values. Do not obtain an Arc<SourceFile> only to calculate an external file identity or selected coordinate.

Tests

Extend the RDR coverage fixture for remapped files, tabs, multibyte input, composed regions, and a private positional edit. Confirm coverage output remains correct and prior dep-info records .spans but no source child.

Root verification

./x fmt --check
./x test compiler/rustc_codegen_llvm
./x test tests/run-make/rdr-coverage-position
./x test tests/incremental/rdr/<coverage-observation-fixture>

Completion gate

Coverage may observe selected anchors in .spans; it may not force exact source loading merely to build its file and coordinate tables.

Phase 6 — Split debuginfo's projected and exact observations

Repository: Rust
Depends on: Phase 5
Purpose: make debuginfo's mixed requirements explicit before exact loading changes.

Implementation

Modify:

compiler/rustc_codegen_ssa/src/mir/debuginfo.rs:77
compiler/rustc_codegen_llvm/src/debuginfo/mod.rs:681
compiler/rustc_codegen_llvm/src/debuginfo/metadata.rs:562
associated debuginfo tests

Use projected identities and anchors for DILocation, scope selection, and line/column reporting. Keep DIFile checksum generation and embedded-source construction on the canonical exact-source path because those operations need file bytes. This is one consumer with two real domain operations, not two aliases for the same operation.

At this phase, exact access still uses the retained serialized SourceFile table. Phase 10 switches that central exact path to the source bundle without changing the consumer boundary established here.

Tests

Separate fixtures must prove:

location-only debuginfo asks only for projected data;
checksum generation takes the exact path;
embedded-source generation takes the exact path;
all three preserve existing object/debug output.

Use dep-info and focused instrumentation to distinguish the paths; do not infer which path ran from flags alone.

Root verification

./x fmt --check
./x test compiler/rustc_codegen_ssa
./x test compiler/rustc_codegen_llvm
./x test tests/run-make/rdr-position-outputs
./x test tests/incremental/rdr/<debuginfo-observation-fixtures>

Completion gate

Line/scope construction must not invoke exact source loading. Checksum and embedding must remain visibly exact so Phase 10 can attach named-child dep-info.

Phase 7 — Capture actually-read Buck sources

Repository: Buck
Depends on: Phase 1; may run in parallel with Rust Phases 2–6
Purpose: produce the durable source bundle without exposing it to downstream actions yet.

Implementation

Modify:

prelude/rust/sources.bzl
prelude/rust/context.bzl
prelude/rust/build.bzl
prelude/rust/tools/rustc_action.py
prelude/cxx/tools/makefile_to_dep_file.py
prelude/cxx/tools/dep_file_utils.py
focused Python/Starlark tests

Retain symlinked_srcs as the compilation input view, but stop treating its single directory artifact as the complete source model. Add one final canonical source-view descriptor to CompileContext containing:

the compilation root already passed to rustc;
the current crate's owned roots/artifacts, including the opaque srcs_filegroup root when used;
the configured physical-to-logical remap rules in their exact precedence order.

Extend source assembly and environment/OUT_DIR processing to contribute their owned artifacts and remap entries to this descriptor. Buck does not currently retain generated, included, and OUT_DIR ownership in one reusable mapping, so this phase creates that canonical shape rather than claiming it already exists.

rustc_action.py already adds execution-time cwd, realpath, and OUT_DIR rewrites. Build one final ordered remap list there, use that same list both for the rustc arguments and for mapping dep-info paths, and apply it exactly once.

Extract the existing Make dependency parser into one reusable implementation instead of writing a second parser with different escaping behavior. After rustc succeeds and writes dep-info, have the same action:

identify the entries owned by the current crate's known source mappings;
map each physical path through the existing logical remapping;
copy the original bytes to <extern-stem>.source-bundle/<logical-key>;
write the sorted logical keys to <extern-stem>.source-bundle-names.

Entries belonging to dependency artifacts or dependency source bundles are not copied into the current crate's bundle. Do not traverse a dependency bundle to make that decision.

The bundle and names manifest are same-action outputs. They are not yet attached to downstream compiler actions in this phase.

Tests

Add focused wrapper tests with synthetic Make dep-info covering:

declared and mapped sources;
generated sources;
include!, include_str!, and include_bytes! inputs;
relevant build-script/OUT_DIR inputs;
escaped spaces and backslashes;
exclusion of upstream .rmeta, .spans, and source-bundle children;
stable logical names across distinct physical action roots.

Root verification

buildifier <changed Starlark files>
buck2 build prelude//rust/tools:scripts-typing
buck2 test prelude//rust/tools:test_rustc_action

Create the focused test target in this phase if it does not yet exist.

Completion gate

Inspect the produced tree and manifest. It must contain exactly current-crate files rustc reported as read, under existing remapped names, without reading or copying transitive source bundles. The one canonical source-view descriptor and the wrapper's one ordered remap list must drive both compilation and capture.

Phase 8 — Model and propagate co-owned Buck sidecars

Repository: Buck
Depends on: Phase 7
Purpose: make illegal RDR artifact combinations unrepresentable throughout Buck's Rust providers.

Implementation

Modify:

prelude/rust/build.bzl
prelude/rust/link_info.bzl
prelude/rust/outputs.bzl
prelude/rust/rust_library.bzl
prelude/rust/failure_filter.bzl
prelude/rust/tools/failure_filter_action.py
provider and wrapper tests

Use one nested sidecar shape equivalent to:

RustArtifactFiles {
    artifact,
    rdr_sidecars: None | {
        spans,
        source_bundle,
        source_names_manifest,
    },
}

Adapt the existing canonical RustArtifact, RustcOutput, EmitOperation, and RustLinkStrategyInfo paths rather than adding a parallel source-aware API. Every metadata-fast, metadata-full, clippy, and link action owns the sidecars it actually emitted.

Extend the existing required-output mechanism with a directory form. On the known infallible compiler-error path, materialize an infrastructure-only empty declared directory so the producing Buck action can report success and reach failure filtering, just as the current wrapper creates missing required files. The failure status must discard the entire sidecar record; the empty directory must never enter RustArtifactFiles, transitive staging, or a consumer action.

On a successful producer, a usable RDR artifact has its artifact, .spans, source bundle, and names manifest. Missing co-owned output fails directly. Do not synthesize successful contents or fall back to artifact-only state.

Tests

Cover every emission variant, provider projection, link strategy, and failure filter outcome. Assert that RDR artifacts cannot expose .spans without the bundle or vice versa, that a failed-action empty directory never reaches a provider, and that non-RDR/proc-macro artifact shapes remain unchanged.

Root verification

buildifier <changed Starlark files>
buck2 build prelude//rust/tools:scripts-typing
buck2 test prelude//rust/tools:test_rustc_action
buck2 test prelude//rust/tools:test_failure_filter_action

Completion gate

Review provider construction sites semantically. There must be one canonical artifact shape and no trio of independently optional fields. Placeholder directory behavior is confined to the known failed-action path and is unobservable after failure filtering.

Phase 9 — Stage direct and transitive source bundles without observing them

Repository: Buck
Depends on: Phase 8
Purpose: make every owner's bundle available under a conventional sibling path while preserving named-child granularity.

Implementation

Modify:

prelude/rust/build.bzl:883
prelude/rust/build.bzl:918
prelude/rust/tools/transitive_dependency_symlinks.py:109
its JSON input/projection and focused tests

Extend the existing artifact/.spans dynamic-name staging record with the source bundle and names manifest. Reuse the current real crate-name and collision-directory selection.

For each transitive artifact, the staging action reads the small names manifest and creates a real nested directory tree of per-child symlinks. It does not open the source bytes and does not traverse the bundle directory. Do not symlink the bundle root as a single directory node.

Keep the original source bundle as the owned artifact behind the staged tree. B's provider may carry A's sidecars transitively, but B's own bundle must never contain A's children.

Do not attach these bundles as broad hidden inputs to a rustc action yet. Phase 12 attaches them atomically with exact depfile selection after rustc can record named children.

Tests

Add a dedicated directory-selector test for the staged symlink shape; the existing generic copied-directory test is not sufficient. Cover:

direct and transitive artifacts;
static and dynamic crate names;
nested logical keys;
collision directories;
A → B → C original ownership;
changing an unselected original child while selecting another staged child.

Root verification

buck2 build prelude//rust/tools:scripts-typing
buck2 test prelude//rust/tools:test_transitive_dependency_symlinks
buck2 test fbcode//buck2/tests/core/executor:test_dep_files

Completion gate

The root review must confirm the staging action reads only manifests, not bundle contents, and that the focused depfile test preserves per-child selection through the symlink tree.

Phase 10 — Load exact sources and remove whole-file data from `.spans`

Repository: Rust
Depends on: Rust Phase 6 and Buck Phase 9's finalized sibling contract
Purpose: make the named source child the only exact-source artifact.

Implementation

Modify:

compiler/rustc_metadata/src/locator.rs
compiler/rustc_metadata/src/rmeta/mod.rs
compiler/rustc_metadata/src/rmeta/encoder.rs
compiler/rustc_metadata/src/rmeta/decoder.rs
compiler/rustc_metadata/src/rmeta/decoder/cstore_impl.rs
compiler/rustc_span/src/lib.rs
compiler/rustc_span/src/source_map.rs
exact-source consumers found by semantic navigation

Associate each RDR extern with its conventional sibling <extern-stem>.source-bundle/, parallel to existing .spans discovery. This is artifact association, not source-name remapping and not a new rustc flag. Keep the associated physical root out of encoded metadata and stable hashing.

At the existing exact EXTERNAL_SPAN_DATA boundary:

resolve the opaque span's original owner and projected source key;
read the owning crate's .spans projection;
open exactly bundle_root.join(projected_key);
insert that physical child path into Session::file_depinfo;
construct and cache one complete external SourceFile from the original bytes using the projected producer-exported StableSourceFileId, the owning CrateNum, and the existing normalization/analysis path;
register that complete file in the sole global SourceMap;
translate the selected source-relative endpoints into its global BytePos range.

Add one canonical deep SourceMap constructor for a source-backed imported file. Factor the real normalization, hash, line, multibyte, and normalization analysis from SourceFile::new, but assign the producer's exported stable ID and external CrateNum before registration. Do not create a current-crate SourceFile and patch it afterward, and do not rely only on an outer (owner, key) cache: SourceMap's stable-ID index must see the correct producer identity so equal logical paths from different crates cannot alias.

The projected source record therefore carries the producer-exported stable source ID as well as its logical bundle key. The canonical constructor and SourceMap registration provide the cache used by snippets, character selection, multiline layout, checksums, embedding, and coarse conversion.

Then remove the serialized SourceFile table and raw global source positions from SpansArtifactRoot. .spans retains only the projected source identities, anchors, layout, and hygiene data established in Phase 3.

Do not call add_external_src, reverse-remap a filename, authenticate bytes against a serialized hash, validate the key, search alternate roots, repair state, or fall back to the old source map. Non-RDR/coarse metadata keeps its existing behavior.

Tests

Use conventional sibling bundles in compiletest and run-make fixtures to cover:

snippets and surrounding-source operations;
multiline rendering and intermediate line lengths;
character selection and span-extension methods;
suggestions;
debuginfo checksums and embedded source;
coarse metadata conversion of an RDR-deferred external span;
CRLF, BOM, tabs, multibyte text, and remapped names;
one physical source read producing one cached complete SourceFile;
equal logical keys from different crates producing distinct correctly owned SourceFiles;
exact dep-info containing the selected physical child;
missing requested child failing directly;
semantic and projected consumers succeeding without opening any child.

Root verification

./x fmt --check
./x test compiler/rustc_span
./x test compiler/rustc_metadata
./x test compiler/rustc_codegen_llvm
./x test tests/run-make/rdr-artifacts
./x test tests/run-make/rdr-clean-rebuild
./x test tests/run-make/rdr-position-outputs
./x test tests/run-make/rdr-symlinked-metadata
./x test tests/incremental/rdr

Completion gate

Inspect .spans encoding and dep-info. No whole-file source property may remain in .spans, and every exact operation must name only the child or children it opened.

Phase 11 — Prove Rust named-child and edge-local invalidation

Repository: Rust
Depends on: Phase 10
Purpose: make the full compiler-side RDR contract executable and reviewable.

Implementation

Add focused physical-revision A → B → C fixtures under tests/incremental/rdr/, using all Phase 1 directives and rustc_expected_metadata_state wherever rustc runs.

Required cases:

A private edit changes one source child while .rmeta and .spans remain byte-identical.
B previously observed that child, so B runs; B's outputs remain identical, so C remains cached.
B previously observed a sibling child instead, so editing the first child leaves B cached.
A selected coordinate changes, so .spans and the edited source child change; a semantic-only B remains cached.
A projected-position B runs on the .spans change without opening the source child.
A semantic edit changes .rmeta; B runs; unchanged B outputs stop the change before C.
A transitive exact span retains A as owner through B and C opens A's named child directly.
Distinct physical revision filenames retain one logical source key.
Proc-macro and hygiene cases preserve their original owner and do not introduce broad source observation.

Every fixture asserts all known .rmeta, .spans, and named-child byte relationships, including artifacts expected to remain the same. Do not assert only the artifact expected to change.

Root verification

./x fmt --check
./x test tests/incremental/rdr
./x test tests/run-make/rdr-artifacts
./x test tests/run-make/rdr-clean-rebuild
./x test tests/run-make/rdr-position-outputs
./x test tests/run-make/rdr-coverage-position
./x test tests/run-make/rdr-hygiene-identity

Completion gate

For every fixture, manually derive the scheduling decision from prior dep-info and measured output bytes. It must match rustc-not-invoked without referring to the source edit's label.

Phase 12 — Activate exact source selection for core Buck rustc actions

Repository: Buck
Depends on: Buck Phase 9 and Rust Phases 10–11
Purpose: attach source bundles to compiler actions without broadening their fingerprints.

Implementation

Modify the existing direct/transitive dependency and depfile paths in:

prelude/rust/build.bzl:883
prelude/rust/build.bzl:918
prelude/rust/build.bzl:1685
prelude/cxx/tools/makefile_to_dep_file.py
focused executor and Rust action tests

Expose each staged source bundle at the conventional sibling path expected by rustc. Add its root to the existing RDR artifact tag used for .rmeta and .spans; do not create a broad second tag.

Register the consumer's converted Make dep-info against that tag in the same change that attaches the bundles. Confirm that a dep-info entry for a staged child selects the corresponding original child through the symlink tree.

Apply this first to ordinary metadata-fast, metadata-full, clippy, and link rustc actions that already use _rustc_emit and depfile filtering. Do not attach source bundles to an action that lacks equivalent depfile registration.

Tests

Add focused Buck fixtures proving:

semantic-only consumers select .rmeta but not .spans or sources;
projected consumers select .spans but no source child;
exact consumers select exactly one named child;
changing an observed child reruns the consumer;
changing an unobserved sibling yields a dep-file cache hit;
direct and transitive dynamic-name staging preserve the same behavior;
B's compile action can expose A's bundle to C without B reading it.

Use buck2 audit dep-files and action execution evidence, not only final output hashes.

Root verification

buildifier <changed Starlark files>
buck2 build prelude//rust/tools:scripts-typing
buck2 test prelude//rust/tools:test_rustc_action
buck2 test prelude//rust/tools:test_transitive_dependency_symlinks
buck2 test fbcode//buck2/tests/core/executor:test_dep_files

Completion gate

The action input graph, converted depfile, and cache behavior must all show named-child selection. A source-bundle directory digest becoming a direct compiler fingerprint fails the phase.

Phase 13 — Finalize normal rustdoc and coverage dep-info after observation

Repository: Rust
Depends on: Phase 10
Purpose: give normal documentation and coverage actions a complete prior observation record before Buck attaches source bundles.

Implementation

Modify:

src/librustdoc/config.rs
src/librustdoc/core.rs
src/librustdoc/lib.rs:945
focused rustdoc and run-make tests

Normal rustdoc already models --emit=dep-info in src/librustdoc/config.rs:319 and src/librustdoc/core.rs:292, but currently writes the file before HTML/JSON rendering at src/librustdoc/lib.rs:967. Rendering can lazily observe .spans and exact source children after that point. Coverage returns at src/librustdoc/lib.rs:962 before the current write.

Configure and prepare the ordinary rustc dep-info output before the work begins, but finalize it only after the last rendering or coverage operation that can request projected or exact external source data. Every normal rustdoc output path that accepts --emit=dep-info must reach this finalization point. Do not snapshot Session::file_depinfo early and do not introduce a rustdoc-specific RDR manifest.

Position-only documentation operations remain projected. Rendering that asks for snippets or exact source records only the named children it actually opens.

Tests

Add HTML, JSON, and coverage fixtures for semantic-only, projected, exact snippet, checksum/embedded source where applicable, remapped paths, and an unobserved sibling. Assert the final Make dependencies exactly and prove that a late renderer observation is present.

Root verification

./x fmt --check
./x test src/tools/rustdoc
./x test tests/rustdoc
./x test tests/rustdoc-ui
./x test tests/run-make/<rustdoc-rdr-depinfo-fixture>

Completion gate

Normal rustdoc and coverage dep-info must be written after their last source observation. An early snapshot or an output path that bypasses finalization fails the phase.

Phase 14 — Activate normal rustdoc and coverage bundles in Buck

Repository: Buck
Depends on: Buck Phase 12 and Rust Phase 13
Purpose: attach bundles to the documentation actions whose exact dep-info surface is now complete.

Implementation

Modify the normal rustdoc and rustdoc-coverage paths around prelude/rust/build.bzl:142-270.

Request the dep-info finalized in Phase 13, convert it with the same canonical Make parser, and register it against the existing RDR artifact tag in the same change that attaches sidecars. Remove broad source visibility only where the new exact source-bundle contract fully replaces it; preserve unrelated deliberate inputs.

Do not change doctest inputs in this phase.

Tests

For normal rustdoc and rustdoc coverage, cover semantic, projected, exact, observed-child, unobserved-sibling, direct, and transitive source observations. Use audit dep-files and action execution evidence.

Root verification

buildifier <changed Starlark files>
buck2 build prelude//rust/tools:scripts-typing
buck2 test prelude//rust/tools:test_rustc_action
buck2 test <focused rustdoc and rustdoc-coverage targets>

Completion gate

Both actions receive source bundles only through exact depfile filtering. Doctest remains untouched and cannot inherit this phase accidentally.

Phase 15 — Aggregate doctest collection and child-rustc dep-info

Repository: Rust
Depends on: Phase 10
Purpose: give the multi-session doctest action one complete Make dep-info record.

Implementation

Modify:

src/librustdoc/config.rs:513
src/librustdoc/doctest.rs:150
src/librustdoc/doctest.rs:576
shared dep-info parsing/writing code and focused tests

Support --test --emit=dep-info=<final-path>. Doctest has several real observation boundaries, so collect all of them:

configure the documentation collection compiler session to emit its own uniquely named Make dep-info after collection completes;
pass a unique dep-info output to every rustc subprocess that compiles a doctest;
include both the merged doctest bundle compilation and its runner compilation when merged mode invokes rustc twice;
after all subprocesses finish, parse and union their exact dependency paths with the collection-session paths;
write one final Make dep-info for the rustdoc action.

Reuse the existing Make format, escaping, and narrow parsed dependency shape. Unique child paths must remain collision-free under parallel doctest execution. Do not add an RDR-only dependency output and do not infer dependencies from the doctest source list.

Tests

Cover standalone and merged doctests, multiple parallel child compilations, compile-fail/no-run variants, the second merged runner compilation, escaped paths, projected observations, exact named children, and an unobserved sibling. Assert both the intermediate inputs and final union.

Root verification

./x fmt --check
./x test src/tools/rustdoc
./x test tests/rustdoc
./x test tests/rustdoc-ui
./x test tests/run-make/<doctest-rdr-depinfo-fixture>

Completion gate

The final doctest action dep-info must cover the collection session and every child rustc process exactly once. A child process without a unique dep-info output or a pre-subprocess finalization fails the phase.

Phase 16 — Activate doctest bundles and verify remaining Buck Rust actions

Repository: Buck
Depends on: Buck Phase 12 and Rust Phase 15
Purpose: replace broad doctest source visibility only after the aggregate dep-info contract exists.

Implementation

Modify the doctest paths around:

prelude/rust/build.bzl:274-295
prelude/rust/build.bzl:433-437
their focused tests

Request and convert Phase 15's final doctest dep-info. Attach RDR sidecars and register that dep-info against the existing artifact tag atomically. Remove the broad transitive RustSourcesTSet input only where exact bundle access fully replaces it, preserving other deliberate doctest inputs.

Clippy should already inherit Phase 12 through _rustc_emit; verify rather than adding a parallel path. Keep proc-macro link artifacts artifact-only while they remain outside the RDR sidecar format, and prove they do not acquire a broad source dependency. Preserve the deliberate RustLinkInfo/RustSources removal in prelude/rust/linkable_symbol.bzl:57.

Tests

Cover standalone and merged doctests, observed and unobserved children, transitive owners, clippy inheritance, proc-macro policy, and linkable-symbol behavior. Use audit dep-files and action execution evidence.

Root verification

buildifier <changed Starlark files>
buck2 build prelude//rust/tools:scripts-typing
buck2 test prelude//rust/tools:test_rustc_action
buck2 test <focused doctest and clippy targets>

Completion gate

Doctest has exact depfile registration before receiving source bundles. Clippy uses the canonical core path, and proc-macro/linkable-symbol policy remains explicitly artifact-only.

Phase 17 — Add the artifact-agnostic Buck A → B → C oracle

Repository: Buck
Depends on: Phases 14 and 16 (which transitively include Phases 11–13 and 15)
Purpose: enforce the North Star end to end and provide the oracle contract for mutation/fuzz testing.

Implementation

Add a dedicated tests/e2e/test_rdr_artifacts.py fixture and target. For every revision or mutation:

snapshot A's .rmeta, .spans, and every named source child independently;
compute A's changed-path set by comparing actual bytes with the previous snapshot;
read B's previously recorded Buck depfile;
predict whether B runs by exact set intersection;
use read_what_ran to assert the prediction;
if B ran, snapshot B's outputs and replace only B's provider-map entry with B's own changed-path set;
retain A's entry for any real transitive A artifact available to C, read C's prior depfile, and predict C from the union of all available provider entries;
assert C's execution and output bytes.

If C did not previously observe an A-owned transitive artifact, A's retained entry has no effect on the intersection. Do not erase a real A → C edge and do not route it through a fictitious B delta.

Required mutations include:

.rmeta changes at A, B reruns, B outputs stay stable, C stays cached;
selected-position changes at A, with both .spans and source bytes measured;
an observed source-child change;
an unobserved sibling source-child change;
addition or removal of an unobserved named child;
a change that affects multiple artifact kinds simultaneously;
public changes that propagate only to the first stable provider edge;
direct and transitive dependencies;
static and dynamic crate names;
failure-filtered producers;
metadata-fast, metadata-full, link, clippy, rustdoc, coverage, and doctest consumers;
the explicit proc-macro policy.

Use buck2 audit dep-files to assert exact selected paths and independently hash each provider output. The reusable oracle accepts artifact snapshots and prior dep-info; a future fuzzer may supply arbitrary edits without teaching the oracle semantic/private/source/span categories.

Root verification

buck2 test fbcode//buck2/tests/e2e:test_rdr_artifacts
buck2 test fbcode//buck2/tests/core/executor:test_dep_files
buck2 test prelude//rust/tools:test_rustc_action
buck2 test prelude//rust/tools:test_transitive_dependency_symlinks

Completion gate

For every mutation, predicted and actual execution must agree using only measured changed paths and prior dep-info. Any test branch based on the intended meaning of the source edit fails the phase.

Final acceptance criteria

The implementation is complete only when:

compiletest can require .rmeta, .spans, and each named source child to be independently same or different;
physical revision files compile under one stable logical source identity;
every RDR test that runs rustc uses rustc_expected_metadata_state where its incremental query/work-product state matters;
an intermediate crate can transport an external span without observing or re-owning its position;
.spans contains no whole-file source property;
projected consumers record .spans but no source child;
exact consumers record only the owning crate's exact named children;
physical source-bundle roots never affect metadata bytes or displayed source identity;
source loading constructs one complete external SourceFile in the existing global SourceMap with the producer's stable source ID and crate ownership;
the canonical Buck source-view descriptor and one ordered remap list drive both compilation and source capture;
Buck bundles exactly the current crate inputs rustc read under existing remapped names;
transitive staging reads manifests but not source bytes;
Buck depfile filtering preserves named-child selection through staged symlinks;
every consumer action that receives a source bundle has exact depfile registration;
a rerun at B replaces only B's outgoing artifact delta; independently observed A-owned transitive artifacts retain their real A → later-consumer edges;
normal rustdoc/coverage finalize dep-info after rendering, and doctest's final dep-info unions collection plus every child rustc invocation;
failed-action placeholder directories never enter successful artifact providers or consumer actions;
A → B → C and direct transitive-owner tests prove that every private edit stops at the first edge where no previously observed provider output changed;
no new remapping flag, compiler .sources artifact, source hash authentication, fallback lookup, repair, or speculative validation was introduced.

davidbarsky/rdr-plan.md

RDR source-bundle implementation plan

Reconstruction status

North star

Intended final design

Trust boundary and deliberate non-goals

Confirmed starting point

Execution protocol

Dependency graph and ledger

Phase 1 — Compiletest's artifact-agnostic oracle

Implementation

Tests

Root verification

Completion gate

Phase 2 — Preserve transitive external-span ownership

Implementation

Tests

Root verification

Completion gate

Phase 3 — Encode the final projected source/location model

Implementation

Tests

Root verification

Completion gate

Phase 4 — Migrate general locations, panic locations, and the public bridge

Implementation

Tests

Root verification

Completion gate

Phase 5 — Migrate coverage to projected locations

Implementation

Tests

Root verification

Completion gate

Phase 6 — Split debuginfo's projected and exact observations

Implementation

Tests

Root verification

Completion gate

Phase 7 — Capture actually-read Buck sources

Implementation

Tests

Root verification

Completion gate

Phase 8 — Model and propagate co-owned Buck sidecars

Implementation

Tests

Root verification

Completion gate

Phase 9 — Stage direct and transitive source bundles without observing them

Implementation

Tests

Root verification

Completion gate

Phase 10 — Load exact sources and remove whole-file data from .spans

Implementation

Tests

Root verification

Completion gate

Phase 11 — Prove Rust named-child and edge-local invalidation

Implementation

Root verification

Completion gate

Phase 12 — Activate exact source selection for core Buck rustc actions

Implementation

Tests

Root verification

Completion gate

Phase 13 — Finalize normal rustdoc and coverage dep-info after observation

Implementation

Tests

Root verification

Completion gate

Phase 14 — Activate normal rustdoc and coverage bundles in Buck

Implementation

Tests

Root verification

Completion gate

Phase 15 — Aggregate doctest collection and child-rustc dep-info

Implementation

Phase 10 — Load exact sources and remove whole-file data from `.spans`