Nix + Composefs: symlink reduction POC

README.md

Nix + Composefs: symlink reduction POC

Work in Progress — POC only

The context: old PCs running Windows 10 that can't upgrade to Windows 11. End-of-life is coming, migration isn't an option. The idea is to replace Windows with NixOS — but these machines have spinning HDDs where boot time is a real concern.

This POC is a first step to understand whether Composefs could help reduce the symlink overhead in the Nix store. I'd like to keep testing the actual impact before drawing any conclusions.

---

The problem

When NixOS builds a user environment, it creates a system-path derivation — a directory full of symlinks pointing back into the Nix store, one per file per package:

/run/current-system/sw/bin/git  →  /nix/store/xxx-git/bin/git
/run/current-system/sw/bin/curl →  /nix/store/yyy-curl/bin/curl
...

On the test machine:

$ find /nix/store/ -type l | wc -l
271037
$ find /nix/store/ | wc -l
951281

~30% of all store entries are symlinks. This ratio is structural to NixOS: symlinks are its primary mechanism for aggregating isolated packages into coherent environments.

The top contributors:

$ find /nix/store -type l -printf '%h\n' \
  | awk -F/ '{print $4}' | sed 's/^[a-z0-9]\{32\}-//' \
  | sort | uniq -c | sort -rn | head -10
 34619 lutris-0.5.18-fhsenv-rootfs
 33159 steam-1.0.0.85-fhsenv-rootfs
 33087 mint-y-icons-1.8.8
 32324 mint-l-icons-1.7.8
 22628 appimage-run-fhsenv-rootfs
 13396 mint-x-icons-1.7.4
  5911 onlyoffice-desktopeditors-9.1.0-fhsenv-rootfs
  5652 openssl-3.6.2-man
  4240 ncurses-6.5
  2304 system-path  (per generation)

The biggest culprits are icon themes (~33k symlinks each, one per icon alias per size) and FHS environment roots. FHS envs (Steam, Lutris, OnlyOffice, ...) recreate a full Linux directory tree in symlinks to make proprietary binaries believe they're running on a standard Linux system — these symlinks are resolved at every cold launch, before the kernel page cache is warm.

On HDD with limited RAM (the typical profile of old Windows 10 machines), this matters: the first launch of an app like OnlyOffice after boot has to resolve ~5900 symlinks with a cold cache, each potentially causing a random seek. system-path adds another ~2300 symlinks resolved at every service startup during boot.

Whether symlinks are actually the dominant bottleneck in these scenarios is something I haven't measured yet.

---

The approach

Composefs stores the directory tree in a compact metadata file (.cfs), with file contents in a separate content-addressed object store. Symlinks in a .cfs are just entries in that metadata file — no separate inodes on disk.

Instead of a buildEnv generating hundreds of symlinks at build time, each package could be mounted as a composefs image and merged with overlayfs.

---

SELinux

Composefs supports xattr labels per file in its metadata, which is relevant for SELinux environments where every file needs a security context. Worth exploring further.

---

This POC

Prerequisites

Add to your NixOS configuration:

environment.systemPackages = [ pkgs.composefs ];

Then rebuild: sudo nixos-rebuild switch.

Requires overlayfs and erofs kernel modules (available by default on NixOS). No extra kernel module needed — mount.composefs is a userspace helper.

Files

File	Description
test.nix	Toy derivations (libfoo, appbar) for basic testing
nar-to-composefs.py	Parses a NAR stream, extracts files into a content-addressed object store, produces a .cfs image
poc-basic.sh	Builds, converts, mounts and runs libfoo+appbar via composefs
poc-no-symlinks.sh	Compares buildEnv (symlinks) vs composefs overlay

Run

bash poc-basic.sh        # basic composition POC
bash poc-no-symlinks.sh  # symlink comparison POC

What poc-no-symlinks.sh shows

Builds hello, curl and git, then creates two unified environments and counts symlinks:

buildEnv (classic) : 260 symlinks
composefs overlay  : 150 symlinks

The 110 missing symlinks are the aggregation symlinks — the ones buildEnv creates to point from a merged directory back into each package:

/nix/store/xxx-demo-env/bin/git  →  /nix/store/yyy-git/bin/git   ← eliminated
/nix/store/xxx-demo-env/bin/curl →  /nix/store/zzz-curl/bin/curl  ← eliminated

With composefs, the overlay merges packages directly — no intermediate symlinks needed. The 150 remaining symlinks are intra-package ones (sbin → bin, lib64 → lib, etc.) that are part of the packages themselves and are stored as compact metadata entries inside the .cfs image, without requiring separate filesystem inodes.

All three binaries run correctly from the composefs overlay.

Next steps

• Measure actual boot time difference on a real HDD
• Test with a full system profile instead of 3 packages
• Understand integration cost with the rest of NixOS

nar-to-composefs.py

import sys
import os
import hashlib
import subprocess

def read_int(stream):
    data = stream.read(8)
    if not data:
        return None
    return int.from_bytes(data, 'little')

def read_bytes(stream, max_len=None):
    length = read_int(stream)
    if length is None:
        return None
    if max_len is not None and length > max_len:
        raise ValueError(f"Length {length} exceeds maximum {max_len}")
    data = stream.read(length)
    padding = (8 - (length % 8)) % 8
    if padding > 0:
        stream.read(padding)
    return data

def read_string(stream, max_len=None):
    b = read_bytes(stream, max_len)
    if b is None:
        return None
    return b.decode('utf-8')

def escape_field(val):
    if val is None or val == "":
        return "-"
    res = []
    b_val = val.encode('utf-8') if isinstance(val, str) else val
    for b in b_val:
        if 32 <= b < 127 and b not in (ord('\\'), ord(' '), ord('=')):
            res.append(chr(b))
        elif b == ord('\\'):
            res.append('\\\\')
        elif b == ord(' '):
            res.append('\\x20')
        elif b == ord('='):
            res.append('\\x3d')
        elif b == ord('\n'):
            res.append('\\n')
        elif b == ord('\r'):
            res.append('\\r')
        elif b == ord('\t'):
            res.append('\\t')
        else:
            res.append(f"\\x{b:02x}")
    return "".join(res)

def convert_nar(nar_path, basedir, image_cfs):
    dump_lines = []
    os.makedirs(basedir, exist_ok=True)

    if nar_path == "-":
        f = sys.stdin.buffer
    else:
        f = open(nar_path, 'rb')

    try:
        magic = read_string(f)
        if magic != "nix-archive-1":
            raise ValueError(f"Invalid magic: {magic}")

        def parse_node(path):
            expect_tag = lambda expected: expect(expected)

            def expect(expected):
                tag = read_string(f)
                if tag != expected:
                    raise ValueError(f"Expected '{expected}', got '{tag}'")

            expect("(")
            expect("type")
            node_type = read_string(f)

            if node_type == "regular":
                is_executable = False
                tag = read_string(f)
                if tag == "executable":
                    empty = read_string(f)
                    if empty != "":
                        raise ValueError("Executable tag has non-empty value")
                    is_executable = True
                    tag = read_string(f)

                if tag != "contents":
                    raise ValueError(f"Expected 'contents', got '{tag}'")

                contents = read_bytes(f)
                expect(")")

                h = hashlib.sha256(contents)
                digest = h.hexdigest()

                prefix = digest[:2]
                rest = digest[2:]
                obj_dir = os.path.join(basedir, prefix)
                os.makedirs(obj_dir, exist_ok=True)
                obj_path = os.path.join(obj_dir, rest)

                if not os.path.exists(obj_path):
                    with open(obj_path, 'wb') as obj_file:
                        obj_file.write(contents)

                mode = "100555" if is_executable else "100444"
                payload = f"{prefix}/{rest}"

                line = f"{escape_field(path)} {len(contents)} {mode} 1 0 0 0 1.0 {escape_field(payload)} - -"
                dump_lines.append(line)

            elif node_type == "directory":
                line = f"{escape_field(path)} 0 40555 2 0 0 0 1.0 - - -"
                dump_lines.append(line)

                while True:
                    tag = read_string(f)
                    if tag == ")":
                        break
                    if tag != "entry":
                        raise ValueError(f"Expected 'entry' or ')', got '{tag}'")

                    expect("(")
                    expect("name")
                    name = read_string(f)
                    expect("node")

                    child_path = "/" + name if path == "/" else path + "/" + name
                    parse_node(child_path)
                    expect(")")

            elif node_type == "symlink":
                expect("target")
                target = read_string(f)
                expect(")")

                line = f"{escape_field(path)} 0 120777 1 0 0 0 1.0 {escape_field(target)} - -"
                dump_lines.append(line)

            else:
                raise ValueError(f"Unknown node type: {node_type}")

        parse_node("/")

    finally:
        if nar_path != "-":
            f.close()

    dump_file = image_cfs + ".dump"
    with open(dump_file, 'w', encoding='utf-8') as df:
        for line in dump_lines:
            df.write(line + "\n")

    print(f"Creating composefs image {image_cfs} from {dump_file}...")
    cmd = [
        "mkcomposefs",
        f"--digest-store={basedir}",
        "--from-file",
        dump_file,
        image_cfs
    ]

    subprocess.run(cmd, check=True)
    os.remove(dump_file)
    print("Composefs image created successfully.")

if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("Usage: python nar-to-composefs.py <nar_file> <basedir> <image_cfs>")
        sys.exit(1)
    convert_nar(sys.argv[1], sys.argv[2], sys.argv[3])

poc-basic.sh

#!/usr/bin/env bash

# Nix + Composefs POC script
set -euo pipefail

echo "=== 1. Building packages with standard Nix ==="
nix build -f test.nix appbar --no-link

PATH_LIBFOO=$(nix-instantiate --eval -E '(import ./test.nix).libfoo.outPath' | tr -d '"')
PATH_APPBAR=$(nix-instantiate --eval -E '(import ./test.nix).appbar.outPath' | tr -d '"')

HASH_LIBFOO=$(basename "$PATH_LIBFOO")
HASH_APPBAR=$(basename "$PATH_APPBAR")

echo "Libfoo: $PATH_LIBFOO"
echo "Appbar: $PATH_APPBAR"

echo -e "\n=== 2. Cleaning old mounts & setting up directories ==="
sudo umount -f "./tmp$PATH_APPBAR" 2>/dev/null || true
sudo umount -f "./tmp$PATH_LIBFOO" 2>/dev/null || true

sudo rm -rf ./tmp
mkdir -p ./tmp/nix/store/.composefs-objects
mkdir -p ./tmp/nix/store/.composefs-meta

echo -e "\n=== 3. Converting NAR archives to Composefs ==="
nix-store --dump "$PATH_LIBFOO" | python3 ./nar-to-composefs.py - ./tmp/nix/store/.composefs-objects ./tmp/nix/store/.composefs-meta/"$HASH_LIBFOO".cfs
nix-store --dump "$PATH_APPBAR" | python3 ./nar-to-composefs.py - ./tmp/nix/store/.composefs-objects ./tmp/nix/store/.composefs-meta/"$HASH_APPBAR".cfs

echo -e "\n=== 4. Creating mount points and mounting ==="
mkdir -p "./tmp$PATH_LIBFOO"
mkdir -p "./tmp$PATH_APPBAR"

echo "Mounting libfoo..."
sudo mount -t composefs "./tmp/nix/store/.composefs-meta/$HASH_LIBFOO.cfs" "./tmp$PATH_LIBFOO" -o basedir="./tmp/nix/store/.composefs-objects"

echo "Mounting appbar..."
sudo mount -t composefs "./tmp/nix/store/.composefs-meta/$HASH_APPBAR.cfs" "./tmp$PATH_APPBAR" -o basedir="./tmp/nix/store/.composefs-objects"

echo -e "\n=== 5. Active composefs mounts: ==="
mount | grep composefs || echo "No composefs mounts found!"

echo -e "\n=== 6. Running the app from Composefs! ==="
echo "Running: ./tmp$PATH_APPBAR/bin/appbar"
echo "----------------------------------------"
"./tmp$PATH_APPBAR/bin/appbar"
echo "----------------------------------------"

echo -e "\nNix + Composefs POC execution completed successfully!"

poc-no-symlinks.sh

#!/usr/bin/env bash
# POC: Nix system-path (symlinks) vs Composefs overlay (no symlinks)
set -euo pipefail

PKGS=(hello curl git)
PKG_ATTRS=(nixpkgs#hello nixpkgs#curl nixpkgs#git)

echo "=== 1. Building packages ==="
for attr in "${PKG_ATTRS[@]}"; do
  nix build "$attr" --no-link
done

declare -A PATHS
for p in "${PKGS[@]}"; do
  PATHS[$p]=$(nix eval nixpkgs#$p.outPath | tr -d '"')
  echo "$p: ${PATHS[$p]}"
done

echo -e "\n=== 2. Classic Nix approach: buildEnv (system-path with symlinks) ==="
ENV_PATH=$(nix build --no-link --print-out-paths --impure --expr "
  let pkgs = import <nixpkgs> {};
  in pkgs.buildEnv {
    name = \"demo-env\";
    paths = [ pkgs.hello pkgs.curl pkgs.git ];
  }
")
echo "buildEnv path: $ENV_PATH"
SL_COUNT=$(find "$ENV_PATH" -type l | wc -l)
echo "Symlinks in buildEnv: $SL_COUNT"
echo "Sample symlinks:"
{ find "$ENV_PATH" -type l | head -8; } > /tmp/sl_sample.txt || true
while IFS= read -r l; do
  echo "  $l -> $(readlink "$l")"
done < /tmp/sl_sample.txt

echo -e "\n=== 3. Composefs approach: overlay of individual packages ==="
mount | grep "$(pwd)/tmp" | awk '{print $3}' | xargs -r sudo umount -f 2>/dev/null || true
sudo rm -rf ./tmp
mkdir -p ./tmp/nix/store/.composefs-objects ./tmp/nix/store/.composefs-meta ./tmp/merged

# Convert each package NAR to composefs
for p in "${PKGS[@]}"; do
  HASH=$(basename "${PATHS[$p]}")
  echo "Converting $p..."
  nix-store --dump "${PATHS[$p]}" | python3 ./nar-to-composefs.py - \
    ./tmp/nix/store/.composefs-objects \
    ./tmp/nix/store/.composefs-meta/"$HASH".cfs
done

# Mount each package via composefs
for p in "${PKGS[@]}"; do
  HASH=$(basename "${PATHS[$p]}")
  mkdir -p "./tmp${PATHS[$p]}"
  sudo mount -t composefs \
    "./tmp/nix/store/.composefs-meta/$HASH.cfs" \
    "./tmp${PATHS[$p]}" \
    -o basedir="./tmp/nix/store/.composefs-objects"
done

# Merge them with a simple overlay (like a composefs-native store would do)
LOWERDIRS=$(for p in "${PKGS[@]}"; do echo -n "./tmp${PATHS[$p]}:"; done | sed 's/:$//')
sudo mount -t overlay overlay ./tmp/merged \
  -o lowerdir="$LOWERDIRS",ro

echo "Merged overlay path: ./tmp/merged"
SL_COMPOSEFS=$(find ./tmp/merged -type l | wc -l)
echo "Symlinks in composefs overlay: $SL_COMPOSEFS"

echo -e "\n=== 4. Comparison ==="
echo "  buildEnv symlinks : $SL_COUNT  (each pointing back into /nix/store)"
echo "  composefs symlinks: $SL_COMPOSEFS  (only intra-package symlinks, no aggregation symlinks)"
echo ""
echo "  Objects in .composefs-objects (shared content store):"
find ./tmp/nix/store/.composefs-objects -type f | wc -l | xargs echo "   "

echo -e "\n=== 5. Run binaries from the composefs overlay ==="
echo -n "hello: "; ./tmp/merged/bin/hello 2>&1 | head -1
echo -n "curl:  "; ./tmp/merged/bin/curl --version 2>&1 | head -1
echo -n "git:   "; ./tmp/merged/bin/git --version 2>&1 | head -1

echo -e "\nPOC completed successfully!"

test.nix

let
  pkgs = import <nixpkgs> {};
in
rec {
  libfoo = pkgs.runCommand "libfoo" {} ''
    mkdir -p $out/lib
    echo "Message from libfoo: Composition successful!" > $out/lib/foo.txt
  '';

  appbar = pkgs.runCommand "appbar" {} ''
    mkdir -p $out/bin
    echo "#!/bin/sh" > $out/bin/appbar
    echo "cat ${libfoo}/lib/foo.txt" >> $out/bin/appbar
    chmod +x $out/bin/appbar
  '';
}