Connor Baker ConnorBaker

Notes

Setup

az account set --subscription "Azure subscription 1"
az configure --defaults group=simpleLinuxTestVMResourceGroup location=eastus
az group create --resource-group simpleLinuxTestVMResourceGroup --location eastus

A note on the numbers

This was run on quiet system with an i9-13900K, 96 GB of DDR5 RAM, and several NVMe SSDs in a ZRAID0 with ZFS. Memory usage never required swapping with hyperfine running two warmups, ZFS ARC and L2ARC effectively cached everything, so IO was not a bottleneck. While it is unlikely there was any thermal throttling as this was largely single-core and temps stayed under 30 C, it is possible.

Additionally, I did not disable boost clocks, set affinity or priority for the process, or other optimizations which could have been done to reduce noise in the results.

They're presented without any liability or guarantee of accuracy, etc.

The workload used isn't representative of a typical Nix evaluation (essentially recursing over Nixpkgs to build a list of lists of attribute paths to all derivations which successfully evaluate), so the numbers aren't indicative of performance each build of Nix would see in practice.

Frustrated with lack of introspection into cost of evaluating a single attribute.

Sad NixOS/nix-eval-jobs#251 is not implemented.

Using https://github.com/ConnorBaker/cuda-packages/tree/main at commit 371d9ee216cde9060a8528c447ee5cc6fd230296:

$ nix eval --allow-unsafe-native-code-during-evaluation --json .#legacyPackages.x86_64-linux.adaCudaPackagesDrvAttrEval | jq

	Running phase: unpackPhase
	@nix { "action": "setPhase", "phase": "unpackPhase" }
	unpacking source archive /nix/store/mp8mmm58idwnlcla8fb2d3289q9m2b26-musl-1.2.5.tar.gz
	source root is musl-1.2.5
	separate-debug-info: recording checksum of source files for debug support...
	setting SOURCE_DATE_EPOCH to timestamp 1709258853 of file "musl-1.2.5/tools/version.sh"
	Running phase: patchPhase
	@nix { "action": "setPhase", "phase": "patchPhase" }
	applying patch /nix/store/mdmbvb93z195n3bij8pa838v3safw346-300-relative.patch
	patching file Makefile

	Running phase: unpackPhase
	@nix { "action": "setPhase", "phase": "unpackPhase" }
	unpacking source archive /nix/store/mp8mmm58idwnlcla8fb2d3289q9m2b26-musl-1.2.5.tar.gz
	source root is musl-1.2.5
	separate-debug-info: recording checksum of source files for debug support...
	setting SOURCE_DATE_EPOCH to timestamp 1709258853 of file "musl-1.2.5/tools/version.sh"
	Running phase: patchPhase
	@nix { "action": "setPhase", "phase": "patchPhase" }
	applying patch /nix/store/mdmbvb93z195n3bij8pa838v3safw346-300-relative.patch
	patching file Makefile

	# To build tests:
	# nix-eval-jobs --force-recurse ./tensorrt-samples.nix --select 'attrs: attrs.tests' \| \
	# jq -cr '.drvPath \| select(.!=null) + "^*"' \| \
	# nix build --keep-going --no-link --print-out-paths --stdin --builders '' -L -j4
	{
	system ? "x86_64-linux",
	cudaCapabilities ? [ "8.9" ],
	cudaVersionComponents ? 2,
	tensorrtVersionComponents ? 2,
	}:

	let
	inherit (builtins)
	add
	div
	elemAt
	isFunction
	length
	mul
	sub
	throw

	# Requires gnu find for -files0-from
	# Requires elfx86exts

	nix path-info -r $1 \
	\| tr '\n' '\0' \
	\| find -L -files0-from - -type f -exec sha256sum {} \+ \
	\| sed -E 's/([a-f0-9]+) (\/nix\/store\/[a-z0-9]{32}-(.+))/\3 \1 \2/' \
	\| sort -s -k 1 \
	\| xargs -n1 -d '\n' bash -c \
	'

	Sourcing python-imports-check-hook.sh
	Using pythonImportsCheckPhase
	source: sourcing cudaHook.bash (hostOffset=-1) (targetOffset=0)
	source: added cudaHookRegistration to prePhases
	source: added cudaSetupCudaToolkitRoot to envHooks for hostOffset=-1
	source: sourcing nvccHook.bash (hostOffset=-1) (targetOffset=0)
	source: added nvccHookRegistration to prePhases
	Sourcing fix-elf-files.sh
	source: sourcing cudaHook.bash (hostOffset=0) (targetOffset=1)
	source: added cudaHookRegistration to prePhases

	nix-eval-jobs --flake .#hydraJobs.sm_89.x86_64-linux --store local --constituents \| \
	jq -cr '.constituents + [.drvPath] \| .[] \| select(.!=null) + "^*"' \| \
	nom build --keep-going --no-link --print-out-paths --stdin