The CI workflow has an 81% failure rate across the last 100 runs. Even on the auto-alignment branch (automated lock file updates, minimal code changes), 78% of runs fail — confirming this is infrastructure flakiness, not code issues.
On the dev branch, only 21 of the last 50 runs succeeded (42% success rate).
Error: × Evaluation error: Failed to get drvPath from shell derivation:
error: path '/nix/store/h9lc1dpi14z7is86ffhl3ld569138595-audit-tmpdir.sh' is not valid
Every CI job resolves devenv via nix build during setup. When the runner's local Nix store is missing a transitive dependency, Nix fails because it doesn't auto-substitute missing store paths during evaluation (only during builds). This hits random jobs non-deterministically depending on the runner's store state.
- Tracked upstream in
effect-utils#201, though that issue is specifically about macOS store corruption. The livestore failures are on Linux (namespace-profile-linux-x86-64) with a different missing path — same class of bug, different manifestation. There is no dedicated issue for the Linux variant in either repo. - The CI already has a repair-path workaround and diagnostic artifact uploads (referencing internal
#272), but they don't reliably fix it. - The
schickling/debug-nix-store-reprobranch (PR #1121, now closed) was a dedicated reproduction attempt.
The web-todomvc-custom-elements example test (set after writing OPFS data) intermittently fails with expect(locator).toHaveValue(expected). Tracked in issue #1062.
Every job is affected roughly equally since the Nix failure happens during setup:
| Job | Failure Rate |
|---|---|
| lint | 53% |
| test-unit | 49% |
| type-check | 44% |
| build-and-deploy-examples-src | 44% |
| perf-test | 41% |
| test-integration-node-sync | 36% |
| build-deploy-docs | 35% |
| wa-sqlite-test | 34% |
| test-integration-playwright (misc/todomvc/devtools) | 13–20% |
| test-integration-sync-provider (various) | 4–17% |
The sync-provider and playwright jobs appear lower mainly because they get cancelled (not counted as failures) when an earlier job already failed.
Comparing 8 successful runs from each period (early January vs. last 2 weeks):
| Job | Jan avg | Recent avg | Change |
|---|---|---|---|
| lint | 2m37s | 6m56s | +165% |
| test-integration-playwright (todomvc) | 2m59s | 5m23s | +81% |
| test-integration-playwright (misc) | 3m22s | 5m49s | +72% |
| test-unit | 3m50s | 6m21s | +65% |
| sync-provider (all variants) | 3m14s | 5m17s | +64% |
| build-and-deploy-examples-src | 9m22s | 12m51s | +37% |
| test-integration-node-sync | 4m33s | 6m07s | +34% |
| test-integration-playwright (devtools) | 5m12s | 6m50s | +31% |
| wa-sqlite-test | 4m57s | 5m22s | +8% |
| build-deploy-docs | 13m48s | 13m41s | -1% |
| perf-test | 18m08s | 11m07s | -39% |
Most jobs slowed by a uniform ~2-3 minutes, consistent with the Nix/devenv setup step taking longer. perf-test is the exception, having improved (likely from code optimization). build-deploy-docs is unaffected.
There are 16 required jobs (same for dev and main branches): lint, test-unit, wa-sqlite-test, test-integration-node-sync, 3 playwright variants, and 9 sync-provider variants. Notably, type-check, perf-test, build-deploy-docs, and build-and-deploy-examples-src are not required.
All 16 run in parallel with no sequential dependencies. Over the last 2 weeks (30 successful runs):
- Median: ~9m30s — when runners are available, everything finishes in 6-10 minutes
- p75: ~25m / p90: ~47m — outliers are caused by runner contention on Namespace runners, not slow tests. In these cases, the bottleneck job itself only takes 4-6 minutes but waits 30-60+ minutes for a runner.
- Typical bottleneck jobs:
test-integration-playwright (devtools),lint,test-integration-node-sync