nvidia-gpu-stack.md

NVIDIA GPU Stack: Capability-Scoped Extension Images

Status: Proposed | Date: 2026-07-21

Problem

The current kata-nvidia-gpu.img monolith includes every NVIDIA library via a broad libnv* glob, sweeping in the full graphics stack (OpenGL, EGL, Vulkan, OptiX, NGX, Wayland) that headless compute workloads never use. Fixing this by shipping separate pre-built images per capability combination leads to a combinatorial explosion of runtimes.

Proposal

Model NVIDIA guest capabilities after NVIDIA_DRIVER_CAPABILITIES: one base image plus per-capability extension images, assembled at pod scheduling time via a pod annotation. The kata runtime reads the annotation and cold-plugs only the requested extensions as guest_extension_images (PR #13285 infrastructure) before VM boot.

Default (no annotation): compute, utility, dcgm

Annotation:

io.katacontainers.config.hypervisor.nvidia_gpu_stack: "compute,utility,dcgm,video"

Capability Map

Capability	Always	Key Contents
compute	yes	libcuda, PTX JIT, NVVM, gpucomp, allocator
utility	yes	libnvidia-ml, nvidia-smi, nvidia-persistenced, nvidia-ctk
dcgm	yes	nv-hostengine, libdcgm
video	opt-in	nvenc, nvdec, cuvid, opticalflow (ffmpeg headless)
nvswitch	opt-in	fabricmanager, libnvidia-nscq, nvlsm (HGX H100/H200)
graphics	never	EGL, GL, Vulkan, NGX, OptiX, Wayland — dropped entirely

Extension Image Layout

/opt/kata/share/kata-containers/
  kata-nvidia-base.img         # kata-agent, NVRC/init, busybox, kmod, iptables, glibc
  kata-nvidia-compute.img
  kata-nvidia-utility.img
  kata-nvidia-video.img
  kata-nvidia-dcgm.img
  kata-nvidia-nvswitch.img

All images are erofs + dm-verity, versioned and shipped as a set by kata-deploy.

Runtime Flow

1. kata-runtime reads nvidia_gpu_stack annotation, falls back to ["compute","utility","dcgm"]
2. Maps each capability to kata-nvidia-{capability}.img by convention
3. Cold-plugs selected images as guest_extension_images before VM boot
4. NVRC discovers, verifies, and mounts extensions (see NVRC Discovery below)

Relation to NVRC

NVRC handles post-boot hardware initialization (module load order, CDI nodes, NVSwitch fabric). Capability selection is done pre-boot by the runtime via annotation. NVRC scope shrinks, decoupling this work from the NVRC PR #167 dependency.

NVRC Discovery

NVRC runs as init in the base image and does not need to be told which capabilities were selected. It discovers them:

1. Scans for attached block devices (/dev/vd*) beyond the base rootfs
2. Verifies each device's dm-verity root hash against the entries in init-data
3. Mounts everything that passes verification
4. Activates the capability (loads kernel modules, starts daemons)

If a device's hash is not in init-data, NVRC refuses to mount it. If a hash is in init-data but no matching device is present, that capability is simply inactive. Kernel params remain static and fully attested. No dynamic data flows through any attested channel.

Confidential Computing (CoCo)

The design holds for confidential workloads without modification by separating two concerns:

Which images are approved: init-data contains the dm-verity root hashes of all capability extension images for a given release. This blob is static per driver version, bound to the base image attestation once at VM launch, and verified by the attestation policy against known reference values. init-data also serves as NVRC's allowlist: only devices whose hashes appear here will be mounted.

Which images are mounted: a runtime decision per pod via annotation. Dynamic, but each mounted image is dm-verity protected against its pre-attested hash. An attacker cannot substitute a malicious extension image because its hash would not match any entry in init-data.

init-data (static per release):
  base.img:     sha256:aaa...
  compute.img:  sha256:bbb...
  utility.img:  sha256:ccc...
  video.img:    sha256:ddd...
  dcgm.img:     sha256:eee...
  nvswitch.img: sha256:fff...

kata-deploy generates this blob as part of the release artifact alongside the images. The attestation verifier checks init-data against the expected hash set for the release. No re-attestation is needed when the pod annotation changes: the full approved set is attested once, subsets are selected freely at runtime.

Phases

Phase	Work	Delivers
1	Replace libnv* glob with per-capability arrays, drop graphics libs	Smaller monolith today
2	Build each capability as a separate erofs image, generate init-data	Extension image artifacts
3	Annotation parsing in runtime-rs + Go runtime, cold-plug selection	Full per-pod composition