Enroot, libnvidia-container, and a path to CDI

Date: 2026-07-24

Sources examined at these versions:

enroot at commit ead3a25 (v4.2.0), /Users/dsrinivas/go/src/github.com/nvidia/enroot
libnvidia-container at commit 3e428194 (v1.19.1 plus 26 commits), /Users/dsrinivas/go/src/github.com/nvidia/libnvidia-container
nvidia-container-toolkit at commit da5f20fb (after v1.19.0), /Users/dsrinivas/go/src/github.com/nvidia/nvidia-container-toolkit

Five research agents produced the underlying reports, and a sixth agent then validated this document against the same commits. The corrections it found are applied. File and line references point at the commits above.

1. Summary

Enroot is a small unprivileged container runtime. It consists of one dispatcher script, four bash libraries, and five small C tools. Its GPU support is one hook script that execs nvidia-container-cli configure from libnvidia-container.

The whole libnvidia-container repository is the legacy injection path. There is no CDI code in it. About 11,900 lines of first-party C and Go sit on top of roughly 255,000 vendored lines plus up to 55,000 lines fetched at build time. The largest block, the Go cgroup library and its vendor tree, is code that enroot never exercises because enroot passes --no-cgroups.

The NVIDIA Container Toolkit has already moved past the legacy path. Since v1.18.0 its runtime defaults to a mode called jit-cdi, which builds a CDI spec in memory at container create time. The CDI stack has no dependency on libnvidia-container, and the packaging reflects that. The one missing piece for enroot is a consumer that applies a CDI spec to a plain root filesystem directory. No such consumer exists anywhere today.

The recommended path is a new enroot hook of roughly 150 to 250 lines of bash and jq. It would read the generated CDI spec, turn the mounts and device nodes into enroot-mount lines, and exec the spec's own nvidia-cdi-hook entries. In parallel, propose an nvidia-ctk cdi apply --root subcommand upstream so the spec interpretation eventually lives in NVIDIA-maintained Go code. A few legacy behaviors do not carry over, and section 6 lists them. For HPC workloads the losses are small.

2. How enroot works

Enroot turns container images into unprivileged sandboxes. It behaves like an enhanced chroot. It uses user and mount namespaces for filesystem separation but deliberately removes most other isolation, which suits HPC clusters where performance and simplicity matter more than isolation.

The pieces:

Piece	Role
`enroot.in`	The CLI dispatcher. The Makefile substitutes `@libdir@`, `@sysconfdir@`, and `@version@` into it.
`src/docker.sh`	Pulls Docker and OCI images layer by layer in parallel and flattens them into a squashfs image.
`src/runtime.sh`	Creates root filesystems from images and starts containers.
`src/bundle.sh`	Builds self-extracting `.run` bundles with the makeself submodule.
`bin/*.c`	Five small C tools built against bundled musl. The main three are `enroot-nsenter`, `enroot-mount`, and `enroot-switchroot`.
`conf/`	Drop-in directories installed to the system config path: `environ.d`, `mounts.d`, and `hooks.d`.

The start sequence runs as follows. enroot start calls runtime::start (src/runtime.sh:294), which execs enroot-nsenter --user --mount to create a user namespace and a mount namespace (src/runtime.sh:364). The user namespace has a single uid map entry, and enroot-nsenter raises ambient capabilities and installs a seccomp filter that fakes success for chown, setuid, setgroups, and similar calls (bin/enroot-nsenter.c:115-152). Inside the namespaces, runtime::_start (src/runtime.sh:221) runs five steps in order:

_do_environ builds one environment file from the image's /etc/environment, the host environ.d directories, the user config, and -e flags, in that order of appending.
_do_mounts_init performs the system mounts, so /proc, /dev, and /sys exist in the root filesystem before hooks run.
_do_hooks executes every executable *.sh in the system and user hooks.d directories in lexical order.
_do_mounts_fini performs the remaining mounts, including any lines hooks appended.
_do_rc prepares the command script, and then the process execs enroot-switchroot to pivot into the root filesystem.

Hooks run inside the namespaces, before the pivot, while the root filesystem is still writable. They see the host filesystem and receive four variables: ENROOT_PID, ENROOT_ROOTFS, ENROOT_ENVIRON, and ENROOT_MOUNTS. A hook that exits nonzero aborts the whole container start (src/common.sh:216). This hook contract is the extension point that GPU support plugs into.

3. How enroot uses libnvidia-container

The entire integration is one hook, conf/hooks/98-nvidia.sh, installed to the system hooks.d (Makefile:52). It does the following:

It imports every NVIDIA_* variable from the container environment file, reading bottom up and skipping keys already set in the process environment (98-nvidia.sh:26-28). The result is a precedence order. Host environment and enroot.conf win over -e flags, which win over environ.d entries, which win over the image's own environment. CUDA images ship NVIDIA_VISIBLE_DEVICES=all, so GPUs appear with no flags at all.
If NVIDIA_VISIBLE_DEVICES is unset or void, the hook exits before touching any dependency (98-nvidia.sh:33-35). A host without a GPU running a plain Ubuntu image never touches the NVIDIA stack. Note the asymmetry with none, which skips only the --device flag and still requires the CLI and driver.
It translates the environment variables into flags. NVIDIA_DRIVER_CAPABILITIES defaults to utility. NVIDIA_MIG_CONFIG_DEVICES and NVIDIA_MIG_MONITOR_DEVICES become --mig-config and --mig-monitor. Each NVIDIA_REQUIRE_* becomes a --require constraint unless NVIDIA_DISABLE_REQUIRE is set.
It fails the container start if nvidia-container-cli is missing (98-nvidia.sh:73-75) and only warns if the nvidia_uvm kernel module is not loaded.
When the compute capability is requested, it bind-mounts /dev/gdrdrv for GDRCopy and the IMEX channels directory itself, using enroot-mount with nofail (98-nvidia.sh:81-84). These two mounts do not involve libnvidia-container.
It execs the CLI (98-nvidia.sh:86):

nvidia-container-cli --user configure --no-cgroups \
    --ldconfig=@/sbin/ldconfig [--device=...] [capability flags] \
    [--require=...] ${ENROOT_ROOTFS}

The three fixed flags match enroot's design. --user runs the privilege separation as the invoking user, which rootless operation requires. --no-cgroups skips device cgroup setup because enroot does not manage cgroups. --ldconfig=@path means the host's ldconfig binary rather than the container's, and libnvidia-container copies it into a sealed memory file descriptor before executing it so the container's own binary is never trusted.

The coupling audit found the claim "one exec" to be substantially true with three precisions. First, the presence check at 98-nvidia.sh:73-75 fails the start just as fatally as the exec two lines later would. Second, the packaging never requires the library. The Debian package only lists libnvidia-container-tools under Suggests (pkg/deb/control:33), and the RPM spec has the equivalent line commented out. Third, two adjacent NVIDIA couplings survive its removal because they use the driver directly. The GDRCopy and IMEX mounts use enroot-mount, and the optional MIG hook depends on nvidia-smi.

Four optional hooks ship disabled under /usr/share/enroot/hooks.d, and two of them touch the NVIDIA stack. 50-mig-config.sh queries nvidia-smi for MIG mode and appends NVIDIA_MIG_CONFIG_DEVICES=all to the environment file when MIG is enabled. 50-sharp.sh sets SHARP and NCCL variables for Slurm jobs. Neither touches libnvidia-container.

4. What libnvidia-container does

The library exists because the NVIDIA driver has a userspace half that must exactly match the kernel module version on the host. Container images can never ship those files, so something must inject them from the host at start time.

The configure flow

nvidia-container-cli configure runs this sequence:

It initializes the library. With --load-kmods and outside a user namespace, a child process chroots to the driver root, loads the kernel modules, and creates device nodes. Inside a user namespace this whole step is skipped (src/nvc.c:250-255), which means enroot's rootless use has always depended on the host having created the device nodes already.
It resolves the container. The root filesystem comes from /proc/<pid>/root, and the library detects Debian versus Red Hat multiarch library directories and globs the container's /usr/local/cuda/compat libraries for forward compatibility.
It discovers the driver. Library discovery parses the binary ld.so.cache on the host against curated lists per capability, e.g. 13 compute libraries and 3 utility libraries (src/nvc_info.c:60-141). It validates each candidate with libelf, checking the driver version suffix and rejecting Mesa libraries that have the same names. Binary discovery walks $PATH for nvidia-smi and the other driver tools. It also finds GSP firmware, device nodes, and IPC sockets.
It evaluates --require constraints with a small expression language supporting comparisons on cuda, driver, arch, and brand values (src/cli/dsl.c).
It mounts everything into the root filesystem (src/nvc_mount.c). The mount categories are:
- a tmpfs mask over /proc/driver/nvidia that exposes rewritten params, version, and registry files
- read-only bind mounts of individual libraries and binaries, filtered by the capability flags
- compatibility symlinks such as libcuda.so
- firmware files, IPC sockets, and device nodes
- per-GPU procfs directories, MIG capability files, and IMEX channel nodes
It refreshes the loader cache. A child process clones new PID and IPC namespaces, pivots into the root filesystem, applies resource limits and a seccomp allowlist of about 60 syscalls, drops to the container owner's uid, and runs ldconfig (src/nvc_ldcache.c:461-563).

The privilege architecture

The library forks helper processes connected over SunRPC on socketpairs, which is why libtirpc is a dependency. The design keeps the proprietary NVML blob out of the privileged parent. The driver child chroots, drops its capabilities to zero, and only then dlopens libnvidia-ml.so.1 to answer queries (src/driver.c:117-173). The parent keeps a 13-capability permitted set and raises specific effective capabilities per phase. In enroot's user namespace these are the capabilities the user already holds over the namespace, so the same code runs rootless.

nvcgo

The Go cgroup library has one job, which is to allow the injected device nodes in the container's device cgroup. On cgroup v1 that is a file write. On cgroup v2 there is no devices controller, so the library queries the attached eBPF device filter program, disassembles it, prepends allow rules, and reattaches it. That eBPF path is why 851 lines of first-party Go carry 246,654 vendored lines, 91 percent of which is golang.org/x/sys. Enroot passes --no-cgroups, so none of this runs for enroot. The service still forks and loads the shared object, but it does no work.

Code size

First-party code:

Component	Lines
Core library C (excluding the vendored nvml.h)	8,720
CLI	2,498
nvcgo first-party files, 851 lines of Go plus build files	988
Build system and packaging	about 2,500
Total	about 14,700

Vendored and fetched code:

Component	Lines
nvml.h, a vendored NVML API header	8,442
nvcgo vendor tree	246,654
nvidia-modprobe utils, fetched at build	2,050
elftoolchain libelf, fetched at build	21,475
libtirpc, fetched at build on modern glibc	31,019

Maintenance signals

The repository is in maintenance mode. The CHANGELOG's newest labeled entry is 1.15.0 rc 2 while git tags continue to v1.19.1, because release notes moved to the toolkit repository, which now carries libnvidia-container as a git submodule and releases it in lockstep. The README still points at the archived nvidia-container-runtime project. Commit volume fell from 178 commits in 2021 to about 32 in 2025, and recent history is mostly dependabot updates. There is no public deprecation notice yet. The legacy stack stays alive precisely for consumers like enroot's hook and Docker's old --gpus path.

5. Where the toolkit is on CDI

CDI, the Container Device Interface, replaces imperative injection with a declarative JSON or YAML spec that describes devices and the container edits they need.

The toolkit's CDI stack is complete and independent:

pkg/nvcdi generates specs with pure Go discovery. It dlopens libnvidia-ml.so.1 through go-nvml and optionally uses libnvidia-sandboxutils. A grep of the CDI paths for nvidia-container-cli finds one hit, a read of the config key that supplies a default driver root (cmd/nvidia-ctk/cdi/generate/generate.go:161). The binary itself is referenced only by the legacy hook and the installer.
Since v1.18.0 the default runtime mode is jit-cdi, which generates the spec in memory at container create time. Legacy mode is only reached when Docker's old --gpus flag invokes nvidia-container-runtime-hook directly. v1.19.0 moved the GDS, MOFED, and CSV modes to jit-cdi as well.
The packaging is already decoupled. nvidia-container-toolkit-base, which ships nvidia-ctk and nvidia-cdi-hook, has no dependency on libnvidia-container. Only the meta package pulls libnvidia-container-tools.
Systemd units named nvidia-cdi-refresh.service and .path regenerate the spec when the driver changes.

A generated nvidia.com/gpu spec contains, per device, the device nodes for that GPU or MIG slice, including DRM nodes found by PCI bus ID. The common edits contain: the four control device nodes such as /dev/nvidiactl; read-only bind mounts for every driver library found in the host loader cache, the driver binaries, IPC sockets, and GSP firmware; graphics configuration files that the legacy path never mounted; two environment variables; and four hooks. The four hooks are create-symlinks, update-ldcache, enable-cuda-compat, and disable-device-node-modification, all invoking the nvidia-cdi-hook binary. A fifth hook, chmod, is deprecated and disabled by default.

Two findings shape the migration design:

No consumer exists that applies a spec to a root filesystem directory. Every consumption path merges edits into an OCI runtime spec through the CNCF CDI library. Nothing anywhere writes a spec's edits into a directory. This is the gap enroot has to fill.
The hook binary works standalone. Each nvidia-cdi-hook subcommand locates the container root by reading a state JSON from stdin that only needs a bundle field, then reading root.path from config.json in that bundle (internal/oci/state.go:33-77). A caller can synthesize both files in a few lines and run the hooks against any root filesystem. update-ldcache re-execs itself into fresh namespaces and pivots into the root filesystem, which requires CAP_SYS_ADMIN in the current user namespace. Enroot's hooks run exactly there, so this works, and it is the same environment rootless podman runs these hooks in.

One more consumption-time detail affects enroot. The spec does not resolve NVIDIA_VISIBLE_DEVICES. Mapping tokens like 0, a GPU UUID, or all onto CDI device names is the consumer's job.

6. What does not carry over

The feasibility work identified seven semantic gaps between the legacy path and a generated CDI spec:

Capability filtering is lost. The legacy path mounts only the library classes requested by NVIDIA_DRIVER_CAPABILITIES, and enroot defaults that to utility. The CDI spec contains everything, and it does not tag mounts by capability, so a consumer cannot rebuild the filtering cleanly. For HPC this mostly means containers see more driver libraries than before.
NVIDIA_REQUIRE_* enforcement is lost. No CDI consumer evaluates the constraint language. A reimplementation in the hook would take roughly 60 to 100 lines of bash and awk, or the ecosystem position that CDI dropped it can be accepted.
The /proc/driver/nvidia mask is lost. Legacy hides unselected GPUs' procfs entries. With enroot's full /proc bind, they stay visible. About 20 lines in the hook can restore the mask for the ENROOT_RESTRICT_DEV case.
MIG config and monitor capability exposure is lost. The spec emits only per-slice access files, not the global mig-config and mig-monitor capability devices, so NVIDIA_MIG_CONFIG_DEVICES and NVIDIA_MIG_MONITOR_DEVICES stop having any effect, and the optional 50-mig-config.sh hook retires.
32-bit compatibility is gone. The CDI discovery reads only the 64-bit half of the loader cache (pkg/lookup/ldcache.go:48). This affects 32-bit graphics stacks and is irrelevant for HPC.
CUDA forward compatibility changes mechanism but not outcome. Legacy bind-mounts the container's compat libraries over the injected ones. CDI writes a loader configuration entry that sorts first. The effective behavior matches, including on musl images.
The graphics application profile tmpfs has no CDI equivalent. This is a niche feature tied to the graphics capability.

One further limit applies to both paths equally. CDI device nodes can carry additional group IDs for nodes that are not world readable and writable, and enroot cannot grant supplementary groups because its seccomp filter fakes setgroups. The legacy CLI cannot grant them either, so nothing regresses.

7. Migration options and plan

The mapping

Every edit in a generated spec maps onto machinery enroot already has:

CDI edit	Enroot primitive
Mounts	One fstab line each through `enroot-mount`. The spec's option strings, `ro,nosuid,nodev,rbind,rprivate` for files and `noexec` variants for sockets, all exist in enroot-mount's option table. `x-create=auto` creates mount points.
Device nodes	Bind mounts from host `/dev`, the exact pattern `98-nvidia.sh:82` already uses for `/dev/gdrdrv`. No mknod is needed, and the legacy path never mknods inside the rootfs either (src/nvc_mount.c:209-245).
Environment variables	Appended to `${ENROOT_ENVIRON}`.
Hooks	Exec the spec's `path` and `args` verbatim, in spec order, with a synthesized state JSON and minimal `config.json`. Order matters, and `enable-cuda-compat` must run before `update-ldcache`.

Device selection is a jq lookup that maps NVIDIA_VISIBLE_DEVICES tokens onto the spec's device names. jq is already a hard enroot dependency on the import path, so the runtime path gains a use of an existing tool rather than a new package.

The options

Option A, a fully self-contained hook that also reimplements the four hooks as file operations, would take roughly 350 to 450 lines. It is feasible but not justified, because the ldconfig logic alone is about 570 lines of subtle Go covering distribution differences, musl, and distroless images, and the set of hooks changes across toolkit versions.
Option B, the recommended one, is a bash and jq hook of roughly 150 to 250 lines that consumes the spec and execs the spec's own hook entries. Bespoke logic is limited to device name resolution, a staleness check, and optionally the NVIDIA_REQUIRE_* evaluation and the procfs mask. New hooks that future toolkit versions add to specs get picked up with zero enroot changes, which already happened once when disable-device-node-modification appeared in v1.17.8.
Option C is contributing nvidia-ctk cdi apply --root <rootfs> upstream. Today nvidia-ctk cdi only offers generate, list, and transform. The building blocks exist, since the hooks already operate on a root filesystem path, and the missing part is applying mounts and device nodes to a directory. Rough size is 500 to 800 lines of Go plus tests and review. With it, enroot's hook shrinks to about 40 lines, and YAML support and future edit kinds come for free. It would also serve other non-OCI consumers such as Apptainer-style runtimes. Section 8 gives the full design.

Option B does not need to wait for option C, and proposing C in parallel moves the spec interpretation to NVIDIA-maintained code over time.

The rollout

Phase 1. Add 97-cdi.sh as opt-in, either via an ENROOT_USE_CDI setting or by detecting a spec file. When it runs successfully it writes a marker so 98-nvidia.sh exits early. No spec means the legacy hook runs as before.
Phase 2. Flip the default so the CDI hook wins whenever a spec is present, and make 98-nvidia.sh log a deprecation warning when it is the one doing the work. Document the gaps from section 6.
Phase 3. Remove 98-nvidia.sh after moving its GDRCopy and IMEX mounts into the CDI hook, change the documented requirement from libnvidia-container-tools to nvidia-container-toolkit-base (doc/requirements.md:57 and the Debian Suggests line), and retire 50-mig-config.sh.

Operational notes

Spec freshness is the main operational risk. The spec embeds versioned paths, so a driver upgrade with a stale spec produces failed binds. The toolkit's refresh units handle regeneration, but they write /var/run/cdi/nvidia.yaml in YAML by default. The hook parses the spec with jq, which needs JSON, so deployments either set NVIDIA_CTK_CDI_OUTPUT_FILE_PATH=/etc/cdi/nvidia.json or the hook shells out to nvidia-ctk cdi generate --format=json when no spec file exists. The hook should stop with a clear error when the spec's driver version disagrees with /proc/driver/nvidia/version.
The refresh units do not watch MIG state. Clusters that reconfigure MIG in a prolog must regenerate the spec there.
Bundles copy the system hooks.d into the self-extracting archive (src/runtime.sh:696), so a bundle built during the transition carries both hooks and behaves the same on the target machine.

8. Option C in detail: the design of nvidia-ctk cdi apply

Goal

Add one subcommand under nvidia-ctk cdi that applies a CDI spec to a root filesystem directory instead of an OCI runtime spec. Runtimes that have no OCI create step, such as enroot, call it once per container start. The command owns spec loading, device selection, mounts, device nodes, environment output, and hook execution. Consumers stop parsing specs themselves, and spec semantics stay in NVIDIA-maintained code.

Command surface

nvidia-ctk cdi apply
    --root PATH        target root filesystem directory (required)
    --device NAME      CDI device name, repeatable, e.g. nvidia.com/gpu=0
                       or nvidia.com/gpu=all
    --spec PATH        explicit spec file; the default is the standard CDI
                       directories /etc/cdi and /var/run/cdi
    --env-output PATH  file to append KEY=VALUE lines to; default is stdout
    --dry-run          print the planned operations and change nothing
    --print-fstab      emit the mounts as fstab lines and the hooks as
                       commands instead of performing them; the caller must
                       apply the mounts before running the hooks

Spec loading and device resolution reuse the CNCF CDI cache, which is what the runtime modifier builds today (internal/modifier/cdi/builder.go:64 and registry.go:46). The command accepts CDI device names, not NVIDIA_VISIBLE_DEVICES tokens. That costs the caller nothing, because the default spec names devices by index, by UUID, and by MIG pair (pkg/nvcdi/namer.go), so the legacy tokens 0, GPU-<uuid>, MIG-<uuid>, and 0:1 are already valid name suffixes. One legacy extra is lost. The legacy CLI also accepts PCI bus IDs as device tokens (src/cli/common.c:186-195), and no CDI name covers those.

A call with no --device flag applies only the spec's common edits. The CNCF cache cannot express that case, because it applies edits only while injecting named devices, so the applier selects the spec and kind itself, with a --kind nvidia.com/gpu flag for hosts that install several specs. The result is close to the legacy meaning of NVIDIA_VISIBLE_DEVICES=none but wider, because the common edits carry every driver library and all four control device nodes while the legacy path filters by capability.

Behavior per edit kind

Mounts. The command performs bind mounts in the caller's current mount namespace, with the option strings the spec carries. It creates missing mount points, as a file or a directory depending on the source, and resolves destination paths inside the root with the same scoped symlink helper the create-symlinks hook already uses. It checks every hostPath before mounting anything and reports all missing paths in one error, because a missing path is the signal that the spec is stale.
Device nodes. As real root the command creates the nodes with mknod, using the recorded major, minor, and file mode. Inside a user namespace mknod is not permitted, so it falls back to bind mounts from the host paths. Bind mounting is also what libnvidia-container does in every case, root included (src/nvc_mount.c:209-245), so the bind path is the proven one and mknod is only a small improvement for the root case. A directory cannot carry the spec's additionalGIDs, so the command warns and continues.
Environment variables. A directory has no environment, so the command appends KEY=VALUE lines to the --env-output file, and enroot points that at ${ENROOT_ENVIRON}. The command drops the spec's NVIDIA_VISIBLE_DEVICES=void entry (pkg/nvcdi/wrapper.go:99) unless asked to keep it. Without that filter the container would see void where the legacy path preserves the image's value, because enroot loads the environment file with the last entry winning and its formatting pass does not remove duplicates.
Hooks. The command executes the spec's createContainer hooks in spec order, with their exact path and args, feeding each one a synthesized state JSON on stdin that points at a temporary bundle whose config.json names the root. That is the full contract the hooks require (internal/oci/state.go:33-77). The command does not reimplement any hook, so hook kinds added in future toolkit versions work without applier changes.

Idempotency and failure

Running apply twice against the same root must be safe. The command reads /proc/self/mountinfo and skips mounts that already exist, and the hooks are already safe to repeat, since symlinks are recreated and the loader cache is regenerated. On failure the command exits nonzero and does not roll back. Rollback has no value here, because the caller aborts the container start and tearing down the namespace removes the mounts.

Privilege model

The command needs CAP_SYS_ADMIN in the current user namespace to mount. A process inside enroot's namespaces holds exactly that, and it is the same environment in which rootless podman runs the CDI hooks today. Real root also works. There is no setuid binary and no file capability.

What the enroot hook becomes

The hook keeps the same environment contract and shrinks to roughly this:

#!/usr/bin/env bash
set -euo pipefail
shopt -s lastpipe
source "${ENROOT_LIBRARY_PATH}/common.sh"

tac "${ENROOT_ENVIRON}" | grep "^NVIDIA_" | while IFS='=' read -r k v; do
    [ -v "${k}" ] || export "${k}=${v}"
done || :

[ "${NVIDIA_VISIBLE_DEVICES:-void}" = "void" ] && exit 0
common::checkcmd nvidia-ctk

devices=()
if [ "${NVIDIA_VISIBLE_DEVICES}" = "all" ]; then
    devices+=("--device" "nvidia.com/gpu=all")
elif [ "${NVIDIA_VISIBLE_DEVICES}" != "none" ]; then
    for d in ${NVIDIA_VISIBLE_DEVICES//,/ }; do
        devices+=("--device" "nvidia.com/gpu=${d}")
    done
fi

exec nvidia-ctk cdi apply --root "${ENROOT_ROOTFS}" \
    --env-output "${ENROOT_ENVIRON}" ${devices[@]+"${devices[@]}"}

The GDRCopy and IMEX mounts from 98-nvidia.sh:81-84, which today apply only when the compute capability is requested, move into this hook, or behind nvidia-ctk cdi apply calls against the gdrcopy and imex-channel spec classes on clusters that generate those specs. They cannot stay where they are, because phase 3 deletes 98-nvidia.sh.

Code layout and sizing

cmd/nvidia-ctk/cdi/apply/apply.go holds the CLI plumbing, registered beside generate, list, and transform (cmd/nvidia-ctk/cdi/cdi.go:44-55). About 150 lines.
A new internal/apply package holds the engine that walks a resolved ContainerEdits value and performs the operations, including the mountinfo check and the state synthesis for hooks. About 250 to 350 lines.
The dry run and fstab printers add about 100 lines.
Tests: golden dry run outputs for synthetic specs, an integration test that runs under unshare and applies a synthetic spec to a temporary root, and one GPU test in the existing end to end suite that generates, applies, and then checks chroot <root> ldconfig -p for libcuda.

The total matches the earlier estimate of 500 to 800 lines of Go plus tests.

Upstream strategy and risks

File the proposal as an issue on nvidia-container-toolkit with this design, and name the other consumers it serves, such as Apptainer and any runtime that prepares a root filesystem without an OCI runtime.
The mount and device node part is vendor neutral and could later move into the CNCF CDI library as a generic helper. Propose in the toolkit first, because the hook execution and packaging questions are NVIDIA specific, and offer the generic part upstream once it settles.
The main risk is that maintainers prefer consumers to go through an OCI runtime. The counterargument is that the hooks already operate on a plain root filesystem path, that jit-cdi shows the team favors CDI consumption everywhere, and that enroot is an NVIDIA project offering to delete its libnvidia-container dependency.

9. Follow-ups

File the upstream issue for nvidia-ctk cdi apply --root using the design in section 8.
Decide whether NVIDIA_REQUIRE_* gets the small bash reimplementation or a documented drop. Modern images rarely rely on it, but Slurm sites may.
Coordinate with pyxis maintainers, since Slurm deployments layer it over enroot and device cgroup enforcement lives there rather than in enroot.
Build the test matrix for the new hook: plain GPU, MIG slices, IMEX channels, GDRCopy, a graphics image, and a musl image for the ldconfig path.

dims/2026-07-24-enroot-libnvidia-container-cdi.md

Select an option

No results found