NixOS/nix#15711 bug reproducers: builtins.storePath + stale stat cache

README.md

PR #15711 bug reproducers

NixOS/nix#15711

Each script is self-contained. Set $NIX to the PR's nix binary (defaults to nix from PATH).

export NIX=/path/to/pr15711/nix
./bug1-storePath.sh
./bug2-nix-copy.sh

Bug 1: builtins.storePath "${self}" fails

prim_storePath calls store->ensurePath() on a lazily-mounted flake path that has not been copied to the store yet. ensurePath tries to substitute, finds nothing, and errors.

Worked before because flakes were eagerly copied to the store.

Fix: call state.ensureLazyPathCopied(path2) before ensurePath() in prim_storePath (src/libexpr/primops.cc).

Bug 2: stale global stat cache → nix copy --impure fails

error: path '/nix/store/...-source/' does not exist

Root cause: PosixSourceAccessor keeps a process-global lstat cache keyed by absolute path (static Cache cache at src/libutil/posix-source-accessor.cc:193).

In impure eval, rootFS is a union of the real filesystem and storeFS. While reading the flake, the union accessor probes the posix layer at /nix/store/<hash>-source, gets ENOENT, and caches that negative result globally. Later ensureLazyPathCopied() materialises the path into the store, but the stale negative entry stays. When Store::narFromPath (the default impl used by UDSRemoteStore) creates a fresh PosixSourceAccessor rooted at that store path, cachedLstat(root) returns the stale nullopt and the dump fails.

Only triggers with --impure (pure eval uses storeFS directly, no posix probe). Re-running succeeds because the path then exists from the start.

Possible fixes:

• have ensureLazyPathCopied() clear the relevant stat-cache entries via PosixSourceAccessor::clearCache(...)/similar after copying, or
• make the storeFS-mounted paths shadow the posix layer so the posix probe is never made for mounted store paths, or
• make UDSRemoteStore::narFromPath go through the daemon instead of reading the local FS (i.e. drop the Store::narFromPath override).

Backtrace (from a debugoptimized build):

#0  nix::SourceAccessor::lstat                     source-accessor.cc:79
#1  ... dumpPath lambda                            archive.cc:53
#2  nix::SourceAccessor::dumpPath                  archive.cc:99
#3  nix::Store::narFromPath                        store-api.cc:366
#4  copyPaths source lambda                        store-api.cc:1074

This likely affects anything that, in a single impure-eval process, materialises a lazy flake path and then reads it back via a PosixSourceAccessor (e.g. nix copy, possibly nix bundle, nix store make-content-addressed, etc.).

Behavior changes (not bugs, intentional)

See behavior-changes.sh.

• nix flake metadata .path may not exist on disk.
• nix-instantiate --eval (read-only by default) prints store paths that do not exist; --read-write-mode materialises them.
• path: flakerefs still copy eagerly; only git:/github:/etc. benefit from lazy copying.

Minor: weak assertion in ensureLazyPathCopied

Only asserts storePath.name() == path.name(), not full equality. If the mounted accessor produced different content than what was hashed at mount time, this would silently copy to the wrong store path and leave the expected one invalid. Probably unreachable with immutable git accessors, but the assertion should be assert(storePath == path).

behavior-changes.sh

#!/usr/bin/env bash
# Intentional behavior changes in PR #15711, shown for reference.

cd "$(dirname "$0")"
. ./common.sh

tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

make_flake "$tmp/flake"
ref="git+file://$tmp/flake"

p=$("$NIX" flake metadata --json "$ref" | jq -r .path)
note "flake metadata .path = $p"
if valid "$p"; then
    note "  exists in store (eager copy / pre-PR behavior or path: flakeref)"
else
    note "  does NOT exist in store (lazy, new behavior)"
fi

note 'nix eval ...#selfPath  (collects context, copies on demand)'
"$NIX" eval "$ref#selfPath"
if valid "$p"; then note "  now copied (expected)"; else bad "  not copied"; fi

# Fresh content for read-only test.
make_flake "$tmp/flake2"
ref2="git+file://$tmp/flake2"
p2=$("$NIX" flake metadata --json "$ref2" | jq -r .path)

note 'nix-instantiate --eval (read-only default): prints path that does not exist'
"${NIX}-instantiate" --eval -E "(builtins.getFlake \"$ref2\").selfPath" 2>/dev/null \
    || "$NIX" eval --read-only "$ref2#selfPath"
if valid "$p2"; then note "  copied"; else note "  not copied (expected in read-only)"; fi

note 'nix-instantiate --eval --read-write-mode: copies'
"${NIX}-instantiate" --eval --read-write-mode -E "(builtins.getFlake \"$ref2\").selfPath" 2>/dev/null || true
if valid "$p2"; then note "  copied (expected)"; else bad "  not copied"; fi

bug1-storePath.sh

#!/usr/bin/env bash
# Bug: builtins.storePath "${self}" fails on lazily-mounted flake source.
#
# prim_storePath calls store->ensurePath() before the lazy path has been
# copied, so it tries to substitute and fails.

cd "$(dirname "$0")"
. ./common.sh

tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

make_flake "$tmp/flake"
ref="git+file://$tmp/flake"

note "nix: $("$NIX" --version)"
note "flake: $ref"

p=$("$NIX" flake metadata --json "$ref" | jq -r .path)
note "flake source store path: $p"
if valid "$p"; then
    note "warning: path already valid, repro may not trigger"
else
    note "path not yet in store (lazy), good"
fi

# shellcheck disable=SC2016 # literal Nix, not shell expansion
note 'evaluating: builtins.storePath "${self}"'
if out=$("$NIX" eval --impure "$ref#storePathSelf" 2>&1); then
    ok "builtins.storePath returned $out"
else
    bad "builtins.storePath failed (regression vs master):"
    printf '%s\n' "$out" | sed 's/^/    /'
fi

bug2-nix-copy.sh

#!/usr/bin/env bash
# Bug: `nix copy --impure --to ... --expr '(getFlake ...).outPath'` fails
# with "path '/nix/store/...-source/' does not exist".
#
# Root cause: PosixSourceAccessor keeps a process-global lstat cache.
# In impure eval, the rootFS union accessor probes the posix layer at
# the (not-yet-existing) store path and caches ENOENT. After
# ensureLazyPathCopied() materialises the path, narFromPath still sees
# the stale negative cache entry and the dump fails.
#
# Pure eval is unaffected (no posix probe). A second run on the same
# content succeeds because the path then exists from the start.

cd "$(dirname "$0")"
. ./common.sh

tmp=$(mktemp -d)
trap 'chmod -R u+w "$tmp" 2>/dev/null || true; rm -rf "$tmp"' EXIT

cache="file://$tmp/cache"
note "nix: $("$NIX" --version)"
note "dest: $cache"

run_copy() {
    local label=$1; shift
    rm -rf "$tmp/cache"
    note "nix copy $* --to $cache --expr '(getFlake ...).outPath'"
    if out=$("$NIX" copy "$@" --to "$cache" --expr "$expr" 2>&1); then
        ok "$label"
    else
        bad "$label"
        printf '%s\n' "$out" | sed 's/^/    /'
    fi
}

# Fresh content per case so the store path is not already valid.
make_flake "$tmp/flake"
ref="git+file://$tmp/flake"
expr="(builtins.getFlake \"$ref\").outPath"
run_copy "impure, fresh content -- expected to FAIL on PR" --impure

note "second attempt on same content (path now valid in store):"
run_copy "impure, path already valid -- expected to PASS" --impure

make_flake "$tmp/flake"
expr="(builtins.getFlake \"$ref\").outPath"
note "pure eval (rootFS = storeFS, no posix probe):"
rm -rf "$tmp/cache"
if out=$("$NIX" copy --to "$cache" "$ref#selfPath" 2>&1); then
    ok "pure mode -- expected to PASS"
else
    bad "pure mode"
    printf '%s\n' "$out" | sed 's/^/    /'
fi

note "control: copying the bare store path (no flake eval) works:"
make_flake "$tmp/flake"
# Force materialisation in a separate process so this process has no stale cache.
p=$("$NIX" eval --raw "$ref#selfPath")
rm -rf "$tmp/cache"
if "$NIX" copy --to "$cache" "$p" 2>&1; then
    ok "nix copy $p"
else
    bad "nix copy $p"
fi

common.sh

# shellcheck shell=bash
set -euo pipefail

: "${NIX:=nix}"

note() { printf '\033[1;34m# %s\033[0m\n' "$*"; }
ok()   { printf '\033[1;32mOK:\033[0m %s\n' "$*"; }
bad()  { printf '\033[1;31mBUG:\033[0m %s\n' "$*"; }

# Create a fresh git flake in a temp dir. Each call commits a unique
# marker so the resulting store path is not already valid.
make_flake() {
    local dir=$1
    rm -rf "$dir"
    mkdir -p "$dir"
    cat > "$dir/flake.nix" <<'EOF'
{
  outputs = { self }: {
    selfPath = "${self}";
    storePathSelf = builtins.storePath "${self}";
  };
}
EOF
    # Unique content so the source store path is fresh each run.
    printf 'marker %s %s %s\n' "$RANDOM" "$$" "$(date +%s%N)" > "$dir/marker.txt"
    git -C "$dir" init -q
    git -C "$dir" add -A
    git -C "$dir" -c user.name=repro -c user.email=repro@localhost commit -q -m init
}

valid() {
    nix path-info "$1" >/dev/null 2>&1
}