Persistent, cross-project Rust CI caching with sccache + Cloudflare R2 (sccache S3 backend, container legs, cold-vs-warm, cross-repo sharing) — with mermaid diagrams

rust-sccache-r2-caching.md

Persistent, cross-project Rust CI caching with sccache + Cloudflare R2

How to stop a Rust release pipeline from recompiling the world on every run — by caching compiled artifacts in S3-compatible object storage (Cloudflare R2), shared across every run and every repo in an org.

Generic writeup of the technique. Replace <ACCOUNT_ID>, bucket names, and secret names with your own.

---

The problem

A release build of a non-trivial Rust app compiles hundreds of crates. In CI that cost lands three times over:

1. Every run is cold. actions/checkout defaults to clean: true (git clean -ffdx), which deletes the gitignored target/. No incremental reuse between runs.
2. Multi-target = ×N. Shipping macOS + Linux + Windows means N full compiles of the same crate graph — one per target triple.
3. Containerized legs are uncached. If some targets build inside Docker (e.g. Linux on a non-Linux host), the cache often doesn't reach inside the container at all, so those legs compile from scratch silently.

The dependency crates — the ~90% you didn't write — are byte-identical run-to-run, and frequently repo-to-repo. Recompiling them every time is pure waste.

---

The idea

sccache is a compiler wrapper (RUSTC_WRAPPER). It caches each compilation unit, keyed by a hash of (preprocessed source + compiler version + flags + target). Point it at an S3-compatible bucket (Cloudflare R2) and that cache becomes:

• Persistent across runs — it lives in R2, not in the workspace, so it survives the target/ wipe.
• Shared across projects — any repo that compiles tokio 1.x for the same target with the same flags gets a hit on what another repo already built.

R2 fits well for two concrete reasons: it speaks the S3 API (so sccache's S3 backend works unmodified), and it has no egress fees (you download cached objects on every build without per-GB charges).

---

How a build resolves each crate

flowchart LR
  cargo["cargo build"] --> wrap["sccache (RUSTC_WRAPPER)"]
  wrap --> q{"key in R2?"}
  q -- hit --> dl["download rlib / object"]
  q -- miss --> rc["compile with rustc"]
  rc --> up["upload result"]
  dl --> link["link / LTO (local)"]
  up --> link
  up -. write .-> r2[("R2 bucket\nS3 API")]
  dl -. read .-> r2

Loading

Only per-crate compilation is cached. The final link (and LTO) still runs locally — but that's a thin slice; the dependency compiles are the bulk.

---

Config

Set the wrapper + S3/R2 backend as job-level env so it's live during the build step:

env:
  RUSTC_WRAPPER: sccache
  SCCACHE_BUCKET: your-cache-bucket
  SCCACHE_ENDPOINT: "https://<ACCOUNT_ID>.r2.cloudflarestorage.com"
  SCCACHE_REGION: auto                  # R2 ignores region; "auto" is the convention
  SCCACHE_S3_KEY_PREFIX: sccache        # optional: namespace within a shared bucket
  SCCACHE_BASEDIR: ${{ github.workspace }}   # path-normalize for cross-run hits
  AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}

Install sccache in the job (e.g. taiki-e/install-action with tool: sccache@<pinned>). Pin the version so the host and any container legs run the same sccache — different versions can compute different cache keys and silently stop sharing.

Confirm it's actually talking to R2 with sccache --show-stats:

Cache location          s3, name: your-cache-bucket, prefix: /sccache/
Cache hits              0
Cache misses            584
Cache errors            0
Average cache write     0.488 s

Cache location: s3 ... + Cache errors: 0 = wired correctly.

---

The gotcha: containerized build legs

If a target builds inside a Docker container (common for Linux on a macOS/ARM host), the host's RUSTC_WRAPPER and credentials do not cross the container boundary — Docker doesn't inherit host env, and the image likely has no sccache binary. Those legs quietly fall back to a full, uncached compile — exactly the legs you most wanted to speed up.

Fix: install sccache in the build image, and forward the config into the container.

# in the linux build image
ARG SCCACHE_VERSION=<pinned>
RUN curl -fsSL "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-<triple>.tar.gz" \
      | tar -xz --strip-components=1 -C /usr/local/bin "sccache-v${SCCACHE_VERSION}-<triple>/sccache"

docker run \
  -e RUSTC_WRAPPER=sccache \
  -e SCCACHE_BUCKET -e SCCACHE_ENDPOINT -e SCCACHE_REGION \
  -e SCCACHE_S3_KEY_PREFIX -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  -e SCCACHE_BASEDIR=/work \
  "$img" bash -lc "cargo build --release --target <triple> && sccache --show-stats"

Run sccache --show-stats inside the container. The host's stats step can't see the container's cache activity — without this you're blind on precisely the legs the fix targets.

---

Cross-project sharing

flowchart TB
  a["repo A — CI"] <--> r2[("shared R2 bucket\nprefix: sccache/")]
  b["repo B — CI"] <--> r2
  c["repo C — CI"] <--> r2
  r2 --- note["first repo to compile a given\ncrate+version+flags+target\npopulates it; the rest hit it"]

Loading

It's not a publish/consume model — there's no producer. Every repo both reads and writes the same bucket; whoever compiles a given unit first fills that entry, everyone else downloads it.

A crate is a shared hit only when the whole key matches: same Rust toolchain, same crate version, same features/flags, same target triple. So the more your repos align (pinned toolchain, overlapping lockfiles), the larger the shared win. Where they drift, each repo still benefits from its own run-to-run caching.

---

Cold vs warm — read the first run correctly

sequenceDiagram
  participant N as Release N — cold
  participant R as R2 cache
  participant M as Release N+1 — warm
  N->>R: every unit misses -> compile + upload
  Note over N,R: ~same wall-clock as before; 0% hit rate is CORRECT
  M->>R: dependency crates -> HIT (download)
  Note over M,R: only your own changed crates recompile
  Note over M: this is where the time drops

Loading

The first release after you enable this is cold: it populates the cache, so expect roughly the same time and a 0% hit rate. That is success, not failure. The payoff is the next release, when the dependency bulk comes straight from R2 and only your own (changed) crates recompile.

A version bump changes only your crates' keys, so even a "release" build hits on the entire third-party dependency graph.

---

Caveats checklist

• Caches compilation, not linking/LTO. Thin/fat LTO defers codegen to link time, which isn't cached — it caps the upside. If you want maximum cache yield, weigh LTO against build time.
• Different target triples don't share with each other (the triple is in the key). The sharing is across runs and repos for the same triple.
• Pin sccache everywhere (host + every container) or keys drift and sharing silently degrades.
• Least-privilege creds. Job-level R2 keys are visible to the build step, which runs third-party build.rs. Use a bucket-scoped token.
• SCCACHE_BASEDIR matters when the absolute build path differs between runs/machines — it strips the prefix before hashing so paths don't break hits.
• Measure, don't assume. sccache --show-stats (hit rate, Cache errors: 0, Cache location: s3 …) is how you confirm it reached the bucket and how you read the warm-vs-cold story.

---

TL;DR

1. RUSTC_WRAPPER=sccache + S3 backend pointed at an R2 bucket (no egress fees, S3-compatible).
2. Wire it into container legs too, and pin the sccache version.
3. First release is cold (populates); subsequent releases hit the dependency bulk.
4. Share one bucket across the org for cross-repo hits — alignment of toolchain/versions/target is what unlocks them.