2024-03-28 Explaining Dependency Offsets And Splicing

2024-03-28-Explaining-Dependency-Offsets-And-Splicing.md

2024-03-28 Explaining Dependency Offsets And Splicing

Most of this is summarized from https://github.com/NixOS/nixpkgs/blob/bc061915ac2cf395e4d1f021cb565f495edfa24a/doc/stdenv/cross-compilation.chapter.md.

Dependency Offsets

In Nixpkgs, dependency offsets are the relative positioning of dependencies in the build process with respect to the build, host, and target platforms. In the case we're not doing cross-compilation, the build, host, and target platforms are the same, and none of this really matters. (Although, with strictDeps = true it can be important as you're telling Nix to enforce separation between build-time and run-time dependencies.)

Some key points:

1. Dependencies are classified into different types based on the platform they run on (build, host, or target) and the platform they generate code for (build, host, target, or none).
2. These dependency types are represented as lists of derivations passed to mkDerivation, with names like deps<host><target>, where host and target can be build, host, or target.
3. Nixpkgs automatically selects dependencies from the appropriate package set based on the dependency type. For example, depsBuildHost dependencies come from pkgsBuildHost, and depsHostTarget dependencies come from pkgsHostTarget.
4. The package sets pkgsBuildHost, pkgsHostTarget, and pkgsTargetTarget are aliases for buildPackages, pkgs, and targetPackages, respectively.
   • This largely due to historical reasons, and is the same reason that we use nativeBuildInputs and buildInputs instead of depsBuildHost and depsHostTarget, respectively.

Some examples:

• A build-time dependency that runs on the build platform and generates code for the host platform (e.g., a compiler) would be a "build → host" dependency. In Nixpkgs, this would be specified using nativeBuildInputs (an alias for depsBuildHost).
• A run-time dependency that runs on the host platform and does not generate code (e.g., a shared library) would be a "host → *" dependency. In Nixpkgs, this would be specified using buildInputs (an alias for depsHostTarget).
• A build-time dependency that runs on the build platform and generates code for the target platform (e.g., a cross-compiler) would be a "build → target" dependency. In Nixpkgs, this would be specified using depsBuildTarget.

Dependency offsets are essentially numerical values that Nixpkgs uses to ensure the correct dependencies are used at each stage of the build process.

By default, the arguments provided by callPackage are from the pkgs package set (an alias to pkgsHostTarget). When you're not cross-compiling, that's fine since the build, host, and target platforms are all the same. However, when you are cross-compiling, that will break stuff -- suddenly the which command in your nativeBuildInputs won't run because its from pkgsHostTarget (aka pkgs) instead of pkgsBuildHost (aka buildPackages), and everything is on fire.

The solution is to accept six different variations of *Packages as arguments for your derivation and carefully select the appropriate dependencies for each one, populating the deps* lists manually.

This is where splicing comes in!

Splicing

Splicing is a technique used in Nixpkgs to combine multiple package sets into a single, unified package set. This is particularly useful for handling cross-compilation, where dependencies need to be drawn from different package sets depending on their role in the build process.

When package sets are built with the splicing utility functions, callPackage "splices" the arguments, allowing some background logic to automatically select the appropriate version of the package based on which dependency list its in -- magic! Well, not really. But close!

When a package defines its dependencies using the lists like depsBuildHost, nativeBuildInputs, depsHostTarget, buildInputs, etc., Nixpkgs automatically selects the appropriate package from the spliced package set based on the dependency offset associated with each list.

For example:

• Dependencies listed in depsBuildHost or nativeBuildInputs will be drawn from the pkgsBuildHost subset of the spliced package set.
• Dependencies listed in depsHostTarget or buildInputs will be drawn from the pkgsHostTarget subset of the spliced package set.

This automatic selection ensures that each dependency is drawn from the appropriate package set based on its intended use in the build process.

Implementation-wise, spliced packages have a __spliced attribute attached to them. So while the argument name provided by callPackage is itself is still fetched from pkgs, the logic for selecting the correct package is based on the __spliced attribute.

As an example, if you wanted to use which in your postPatch phase (which runs on the build platform), you'd want to make sure the version of which you're using is from pkgsBuildHost (aka buildPackages). So you'd need to do something like this:

# All these arguments come from `pkgs` (aka `pkgsHostTarget`) and have the `__spliced` attribute when cross-compiling
{which, stdenv}: 
stdenv.mkDerivation {
  # Snipped for brevity
  postPatch = ''
    # This will fail if we're cross-compiling because the build platform doesn't match the host platform
    # NVCC_PATH="$(${which}/bin/which nvcc)"
    # This will work because we're using the version of `which` from `pkgsBuildHost` (aka `buildPackages`)
    NVCC_PATH="$(${which.__spliced.buildHost}/bin/which nvcc)"
  '';
}