A caching and persistent Nix shell with direnv

.00readme.org

Cached and Persistent Nix shell with direnv integration

Problem being solved

• the default direnv Nix integration from projects loads a Nix shell every new terminal or every time shell.nix and/or default.nix changes. On larger projects with a lot of Nix shell dependencies, that can cause the terminal to take more than 6 seconds to load, which significantly degrades the developer experience (DX).
• when a developer garbage collects in their Nix store, often the Nix shell dependencies are deleted also which causes a slow start the next time the project’s Nix shell is requested, also degrading developer experience (DX).

Solution outline

• on first Nix shell load, the Nix shell is evaluated fully such that direnv dumps the environment from the spawn Nix shell process
• on subsequent loads, a cache directory is inspected to see if the shell was already loaded, and applies the environment settings from the original loading of the Nix shell evaluation
• to determine freshness of shell evaluation we update the nixexprs array in the Bash script above to ensure (see TODO comment in the .direnvrc file), which will watch for changes in that array; these are all used to determine the hash of the cached Nix shell evaluation to ensure consistency
• to ensure all important GC roots are added to prevent premature GC-ing of the shell’s dependencies, we add roots of all indirects for the shell’s derivation

Caveats

It has two main caveats right now (for the infrequent case where you manually GC Nix packages from your store, ~every few months):

1. to free up old packages no longer referenced in the Nix shell, you need to rm -rf .direnv.d in your dailykos project root then direnv reload and then nix-collect-garbage -d
2. after collecting the nix “garbage”, you will then rm -rf .direnv.d and then it should take a few seconds only to pull in some bash doc and dev packages that aren’t creating a GC root under .direnv.d (I am out of ideas why it’s just those two packages).

How to use

• put the .direnvrc file in your $HOME directory.
• put the .envrc (and customize as necessary) to your project’s root, adjacent to shell.nix and default.nix.

Related works

• the direnv project wiki contains similar use_nix overrides but numerous bugs existed based on my requirements, so I put this together to aid my developer experience. Check them out for yourself to compare: https://github.com/direnv/direnv/wiki/Nix

.direnvrc

# Put this file at ~/.direnvrc
hash_env() {
  if has shasum; then
    for e in "${@}"; do shasum -a 256 "${PWD}/${e}"; done | shasum -a 256 | cut -c -64
  else
    fail "do not have shasum to cache environment"
  fi
}

load_nix() {
  local -r orig_IN_NIX_SHELL="${IN_NIX_SHELL:-}"
  local -ra envhash="${1}"
  local -r envdrv="${envdir}/drv"

  nix-shell --show-trace --run 'direnv dump' > "${envdir}/dump"
  direnv apply_dump "${envdir}/dump" > "${envdir}/vars"

  if [ -z "${orig_IN_NIX_SHELL:-}" ]; then
    unset IN_NIX_SHELL
  else
    export IN_NIX_SHELL="$orig_IN_NIX_SHELL"
  fi

  # persist if hash isn't empty
  if [ ! -z "${envhash}" ]; then

    env IN_NIX_SHELL=1 \
      nix-instantiate \
      --add-root "${envdrv}" \
      --indirect "shell.nix" \
      > /dev/null
    nix-store \
      -r $(nix-store --query --references "${envdrv}") \
      --add-root "$(cache_file "${envhash}")/result" \
      --indirect \
      > /dev/null
  fi
}

cache_file() {
  echo "${PWD}/.direnv.d/env-${1}"
}

use_nix() {
  # TODO change list of files that should be watched by direnv to ensure shell consistency at run-time.
  local -ra nixexprs=(shell.nix default.nix version.nix nixpkgs.nix etc/shellHook)
  local -r hash="$(hash_env ${nixexprs[@]})"
  local -r envdir="$(cache_file "${hash}")"
  mkdir -p "${envdir}"

  if [ -z "${hash:-}" ]; then
    load_nix "${hash}"
  else
    if [ -e "${PWD}/.direnv.d/env-${hash}/drv" ]; then
      log_status "using cached environment (${hash})"
    else
      load_nix "${hash}"
    fi
  fi

  for e in "${nixexprs[@]}"; do
    watch_file "${e}"
  done
  watch_file "${direnv}/drv"
  watch_file "${direnv}"

  source_env "${envdir}/vars"
}

.envrc

# sample .envrc file for your project that contains shell.nix, etc

# if you have a .env that needs to be loaded by dotenv you should include this, otherwise remove these two lines
dotenv

# this is the critical peice
use nix

Is there any advantages against https://github.com/target/lorri ?

I had to edit https://gist.github.com/mbbx6spp/731076cb8fc620b064b8e5b28fb1c796#file-direnvrc-L12 to remove the read-only parameter to make it work (I use darwin). Not sure if it's expected

  if [ ! -z "${hash}" ]; then

on line 25, I guess that was meant to be ${envhash}, not ${hash} ?

Is there any advantages against https://github.com/target/lorri ?

It eliminated the need to install a heavyweight dependency that does something very simple.

I had to edit https://gist.github.com/mbbx6spp/731076cb8fc620b064b8e5b28fb1c796#file-direnvrc-L12 to remove the read-only parameter to make it work (I use darwin). Not sure if it's expected

This is due to Darwin shipping a 12+ year old version of bash.

on line 25, I guess that was meant to be ${envhash}, not ${hash} ?

Yes, thank you. Fixing for posterity even though this is now already folded in to nix-direnv and offered implicitly by Nix Flakes.

:)

PSA: Nobody should be using the above shared in Sept 2018!