Setting up a Nix binary cache

content.slab

include layout-page.slab

let titl = "Setting up a Nix binary cache"
let date = "2024-06-05"

page{titl, date}

  style.
    pre, code {
      max-width: 100%;
      overflow-x: auto;
    }

  .switcher
    .flow-all
      p In this text I'm documenting the creation of a Nix binary cache using an S3-compatible object storage, push the closure of a derivation to it, and pull it back. It is not complicated and there is documentation online, but small mistakes are not easy to troubleshoot.

    div
      -- empty

  p When using Nix or NixOS, the official binary cache <code>cache.nixos.org</code> is enabled by default. This is where the prebuilt packages come from when running a command like <code>nix-shell -p hello</code>.

  p When building and distributing your own work, a binary cache can be used to enhance various development and deployment workflows, for instance to enable a better continuous integration experience.

  p In my case, I wanted to document how to provision a binary cache that could be used by the <a href="https://github.com/hypered/thebacknd"><code>thebacknd</code></a>. <code>thebacknd</code> is a small proof-of-concept program to spin a virtual machine in the cloud using a single command, and it relies on a binary cache to store and distribute the system to deploy.

  p I'm going to show commands and syntax used for two different object storages, DigitalOcean Spaces and Backblaze B2, and they should apply similarly to any S3-compatible service such as AWS S3 and MinIO.

  p As a summary:

  ul
    li
      p I assume we're working on a local development machine where we can use Nix.
    li
      p We'll create an S3-compatible bucket where we can store files.
    li
      p We'll create signing keys to sign files in the local Nix store.
    li
      p We'll push signed files from our local Nix store to our bucket.
    li
      p We'll retrieve them. In <code>thebacknd</code> scenario mentioned above, this is done in a new virtual machine to retrieve a new NixOS toplevel to activate.

  h2 Bucket creation and endpoints

  p Data in an object storage are held in resources called buckets. A bucket can be created manually through a web interface as it is done only once. Normally, this can also be done using the standard <code>s3cmd</code> command-line tool, but this doesn't seem possible fo B2: access tokens are associated to buckets, which need to exist before creating a token. This is less developer-friendly, but it also means that access tokens can have better (more restricted) access rights than on DigitalOcean (at the time of writing, DigitalOcean experiments (in beta) retricted access rights but I haven't tried them).

  p Here is the link to the page <a href="https://cloud.digitalocean.com/spaces/new">to create a bucket on DigitalOcean</a>. And here is the link <a href="https://secure.backblaze.com/b2_buckets.htm">to create a bucket on Backblaze</a>.

  p I create a new bucket named "demo-store". (I'm using the AMS3 region, for DigitalOcean. For B2, there is no region to select (the region is bound to the account when it is created) and I keep it private, with no default encryption and no object lock.)

  p If you prefer to create the bucket using the command-line, the section <a href="#bucket-creation-with-s3cmd">Bucket creation with s3cmd</a> below contains examples.

  p After creation, we can view the endpoint used for programmatic access to the buckets: <code>demo-store.ams3.digitaloceanspaces.com</code> and <code>s3.eu-central-003.backblazeb2.com</code> in my case. Note how the bucket name we choose appears as a prefix in the DO case, but not in the B2 case. When we'll reference the endpoints below, mentioning or not the name of the bucket, and where in the <code>s3://</code> URL scheme, will matter.

  h2 Access keys

  p For programmatic access to the buckets, in addition to the endpoints mentioned above, we also need credentials in the form of a pair access key id (public) and secret access key (private).

  p To create keys for DigitalOcean, in <a href="https://cloud.digitalocean.com/account/api/spaces">the API page of the web interface</a>, there is a dedicated tab for Spaces.

  p For Backblaze, they can be created in <a href="https://secure.backblaze.com/app_keys.htm">the Application Keys page</a>. The nice thing here is we can have a per-bucket access/secret key pair, and we can also create a read-only pair.  We need to click the "Add a New Application Key" button, choose <code>demo-store</code> for the name, and we select to allow access to only the <code>demo-store</code> bucket as read-and-write. We can leave the other options empty.

  h2 <code>s3cmd</code> configuration

  p The standard way to interact with an S3 bucket from the command-line is to use the <code>s3cmd</code> tool. Normally, <code>s3cmd</code> uses a configuration file located at <code>~/.s3cfg</code> and will complain if it doesn't exist:

  pre
    code.
      $ s3cmd mb s3://demo-store
      ERROR: /home/thu/.s3cfg: None
      ERROR: Configuration file not available.
      ERROR: Consider using --configure parameter to create one.

  p (<code>mb</code> stands for "Make bucket".)

  p We need to create the configuration file, but instead of using its default location, we pass explicitely the filename we want with the <code>-c</code> option. This allows us to have different configurations for both DigitalOcean and B2.

  pre
    code.
      $ s3cmd --configure -c s3-config-do
      $ s3cmd --configure -c s3-config-bb

  p When prompted, we don't enter the access key and the secret key, and we rely instead on environment variables provided by a <code>.envrc</code> file ([see below](environment-variables)). I think the "region" is only important when using AWS S3 only and I keep "US".

  p For the S3 endpoint, in the case of DigitalOcean, I'm using <code>ams3.digitaloceanspaces.com</code>, and for the DNS-style template, I'm using <code>%(bucket)s.ams3.digitaloceanspaces.com</code>.

  p For Backblaze, we took note above of the endpoint <code>s3.eu-central-003.backblazeb2.com</code> and use it for both the endpoint and the DNS template.

  p For everything else, I use blank or the default value.

  h2 Environment variables

  p To populate environment variables used by <code>s3cmd</code>, I create a <code>.envrc</code> file to be used with the [direnv](https://github.com/direnv/direnv) tool. To use both DigitalOcean and Backblaze, I actually create two such files, in two different directories. It looks like this:

  pre
    code.
      $ cat .envrc
      export AWS_ACCESS_KEY_ID=xxxx
      export AWS_SECRET_ACCESS_KEY=xxxx

  p You can confirm it works with e.g. (this should simply return with no error message):

  pre
    code.
      $ s3cmd -c s3-config-do ls s3://demo-store
      $ s3cmd -c s3-config-bb ls s3://demo-store

  h2#bucket-creation-with-s3cmd Bucket creation with s3cmd

  p If we want to use the command-line to create the bucket in the DigitalOcean case, we can use the <code>mb</code> subcommand. I also show the subcommands to list the content of the bucket, add a file to it, and remove the file.

  pre
    code.
      $ s3cmd -c s3-config-do mb s3://demo-store
      $ s3cmd -c s3-config-do ls s3://demo-store
      $ s3cmd -c s3-config-do put README.md s3://demo-store/some/path/
      $ s3cmd -c s3-config-do ls s3://demo-store --recursive
      $ s3cmd -c s3-config-do del s3://demo-store/some/path/README.md

  h2 Signing keys

  p Our goal is to send a Nix build artifact located in our local Nix store into the bucket. Before doing so, we want to create signing keys to sign our store content (at least the part we want to push to the bucket), so that we can verify later when we pull that content that it was indeed created by us.

  p We can create a public/private key pair used to verify and sign store path with the following command:

  pre
    code.
      $ nix-store --generate-binary-cache-key \
        demo-store \
        cache-priv-key.pem \
        cache-pub-key.pem

  p <code>demo-store</code> is the name of the key. There is no obligation to have it the same name as the bucket, but it helps to remember what it is used for.

  p The public key can be used with the <code>--option substituters</code> and <code>--option trusted-public-keys</code> options of the <code>nix-xxx</code> and <code>nix xxx</code> commands. The private key is used to sign store paths in the next section.

  h2 Signing and uploading to the binary cache

  p To make sure we're actually using our own cache reliably, we need to build something on our own (and not for example something that alreaydy exists in <code>cache.nixos.org</code>) otherwise we might get (and then push) the <code>cache.nixos.org-1</code> signature, or Nix might find what we want later elsewhere that in our own cache.

  p We use the following command to "build" a file in the Nix store, with "asdf" as its content.

  pre
    code.
      $ nix-build -E 'with import <nixpkgs> {}; writeText "example.txt" "asdf"'

  p Then we sign it with the private key generated in the previous section:

  pre
    code.
      $ nix store sign --recursive --key-file cache-priv-key.pem $(readlink result)

  p We can use the <code>nix copy</code> command to upload our build artefact. This also copies the signature. Then we can use <code>s3cmd ls --recursive</code> to check we have indeed some files in the bucket.

  p For the DigitalOcean case, note that we include the bucket name in the endpoint:

  pre
    code.
      $ nix copy \
        --to "s3://cache?endpoint=demo-store.ams3.digitaloceanspaces.com" \
        $(readlink ../result)
      $ s3cmd -c s3-config-do ls s3://demo-store --recursive

  p And for the Backblaze case, we don't.

  pre
    code.
      $ nix copy \
        --to "s3://demo-store/cache?endpoint=s3.eu-central-003.backblazeb2.com" \
        $(readlink ../result)
      $ s3cmd -c s3-config-bb ls s3://demo-store --recursive

  p If you're curious, you can list the files in the bucket and download the <code>.narinfo</code> one. You should see a line starting with <code>Sig: demo-store:</code> followed by the signature.

  p Note: it may happen that you forget or fail to properly sign a store path, then push it. In that case, I don't know of a proper way to force the upload of the signature afterwards. Instead, the file should be deleted and reuploaded When the file is not signed, this looks like this:

  pre
    code.
      error: cannot add path '/nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt'
      because it lacks a signature by a trusted key

  h2 Downloading from the binary cache

  p When you built the <code>example.txt</code> file above, a Nix store path was produced. In my case:

  pre
    code.
      $ nix-build -E 'with import <nixpkgs> {}; writeText "example.txt" "asdf"'
      /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt

  p Before we can download it, we need to remove it from our local Nix store. As root (because of <a href="https://github.com/NixOS/nix/issues/6141">this issue</a>):

  pre
    code.
      # nix store delete /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt \
        --ignore-liveness

  p We can also use <code>nix copy</code> to download the store path. For the DigitalOcean case:

  pre
    code.
      $ nix copy \
        --from "s3://cache?endpoint=demo-store.ams3.digitaloceanspaces.com" \
        --option trusted-public-keys $(cat ../cache-pub-key.pem) \
        /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt

  p In the Backblaze case:

  pre
    code.
      $ nix copy \
        --from "s3://demo-store/cache?endpoint=s3.eu-central-003.backblazeb2.com" \
        --option trusted-public-keys $(cat ../cache-pub-key.pem) \
        /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt

  p Note: the following command also work when used as root, but not as a regular user. Te required credentials that we set up as environment variables are not properly passed to the Nix daemon doing the download.

  pre
    code.
      # nix-store -r /nix/store/s20n2999afk451sa4s4gyx4q7x9vsx8x-example.txt \
        --option substituters "s3://cache?endpoint=demo-store.ams3.digitaloceanspaces.com" \
        --option trusted-public-keys $(cat ../cache-pub-key.pem)

layout-page.slab

frag page{titl, date}
  doctype html
  html(dir="ltr", lang="en")
    head
      meta(charset="utf-8")
      meta(name="viewport" content="width=device-width, initial-scale=1")
      link(rel="stylesheet", href="/static/css/struct.css")
      title= titl
    body.u-container-vertical
      header
        .u-container
          .u-bar
            .u-bar__left
              style.
                .a-logo { color: black; text-decoration: none; }
              a.a-logo(href="/")
                div
                  span noteed
            .u-bar__right
              -- empty

      main
        .u-container.u-flow-c-4
          .c-content.flow-all.u-flow-c-4.limit-42em
            h1= titl
            small.breadcrumb= date

            content