You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
nvidia-smi works, shows 8x NVIDIA GB200 NVL with correct attributes
NVML queries work (name, UUID, memory, architecture, compute capability)
CDI injection works (nvidia-smi available inside containers)
ResourceSlices published: Blackwell, cc 10.0, 192Gi, unique UUIDs/PCI bus IDs
Both gpus and compute-domains containers 2/2 Running
No real CUDA compute (mock CUDA library stubs only)
No MIG support (returns NOT_SUPPORTED)
No real NVLink fabric or IMEX channels
Driver root at /var/lib/nvml-mock/driver (not /)
Mock device nodes at /var/lib/nvml-mock/dev/nvidia* (not /dev/nvidia*)
Test pods use ubuntu:24.04 image, run nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait
Test-by-Test Verdict
test_gpu_basic.bats -- 5/5 YES
#
Test
Tags
Compatible
What it does
1
1 pod, 1 full GPU
fastfeedback
YES
nvidia-smi -L in pod, checks UUID: GPU-
2
2 pods, 1 full GPU each
fastfeedback, multi-gpu
YES
Allocates 2 GPUs, compares UUIDs differ
3
2 pods, shared 1 GPU (1 RC)
fastfeedback
YES
Shares GPU via ResourceClaim, UUIDs match
4
1 pod, 2 containers, shared GPU (1 RCT)
fastfeedback
YES
Same GPU visible in both containers
5
inspect device attributes in resource slice
fastfeedback, version-specific
YES
Checks ResourceSlice attribute keys
All tests use ubuntu:24.04 with nvidia-smi -L. No CUDA, no MIG, no special hardware.
test_gpu_robustness.bats -- 7/8 YES, 1 CAVEAT
#
Test
Tags
Compatible
What it does
8
kubelet-plugin exposes Prometheus metrics
fastfeedback, gpu-robustness
YES
Curls :8080/metrics, checks for nvidia_dra_prepared_devices
9
ResourceClaim lifecycle
fastfeedback, gpu-robustness
YES
Create pod -> allocation has "devices" -> delete -> no orphans
10
GPU re-acquired after pod delete/recreate
fastfeedback, gpu-robustness
YES
Delete pod, recreate, check UUID again
11
pod with 2 RCTs gets 2 distinct GPUs
fastfeedback, gpu-robustness, multi-gpu
YES
Count UUID: GPU- lines >= 2
12
CEL selector matches GPU by architecture
fastfeedback, gpu-robustness
YES
Selector matches "Blackwell" from ResourceSlice
13
CEL selector rejects non-matching productName
fastfeedback, gpu-robustness
YES
Pod stays Pending (correct)
14
ResourceSlice device count matches host GPU count
fastfeedback, gpu-robustness
CAVEAT
Mounts host /dev, counts nvidia[0-9]*
15
rapid claim create/delete no resource leak
fastfeedback, gpu-robustness
YES
10 rapid create/delete cycles
Caveat on test #14: Creates a privileged pod that mounts host /dev and runs
ls /host-dev/nvidia[0-9]* | wc -l. Our mock devices live at /var/lib/nvml-mock/dev/,
not /dev/. The test will see 0 devices and fail. Fix options:
Create symlinks at /dev/nvidia* -> /var/lib/nvml-mock/dev/nvidia*
Or skip this test with a new tag
test_gpu_sharing.bats -- 1/2 YES, 1 MAYBE
#
Test
Tags
Compatible
What it does
6
TimeSlicing -- 2 containers share GPU
fastfeedback, gpu-sharing
YES
2 containers, same UUID, nvidia-smi only
7
MPS -- 2 containers share GPU
fastfeedback, gpu-sharing
MAYBE
MPS control daemon must start; uncertain without real GPU
MPS test deploys an mps-control-daemon pod. The daemon may crash without a real GPU
to bind to. Would need to test empirically.
test_gpu_extres.bats -- 1/1 YES
#
Test
Tags
Compatible
What it does
16
legacy nvidia.com/gpu: 1 with DRAExtendedResource
extres, fastfeedback
YES
k8s 1.35+ feature; creates pod with resource limit, verifies auto-created ResourceClaim
Requires DRAExtendedResource=true feature gate (enabled in our Kind config for k8s 1.35).
test_gpu_cuda_workloads.bats -- 2/4 YES, 2 NO
#
Test
Tags
Compatible
What it does
17
CUDA demo suite (deviceQuery, vectorAdd, bandwidthTest)
fastfeedback, gpu-workloads
NO
Uses nvidia/cuda:12.5.0-devel-ubuntu22.04, compiles+runs CUDA samples
18
Job with RCT, 2 completions
fastfeedback, gpu-workloads
YES
Job scheduling, nvidia-smi UUID check
19
Deployment 2 replicas, distinct GPUs
gpu-workloads, multi-gpu
YES
Parallel allocation, nvidia-smi UUID check
20
busGrind memory bandwidth stress
gpu-workloads, gpu-busgrind
NO
CUDA memory stress benchmark
Tests #18 and #19 are in the same file as CUDA tests but don't use CUDA -- they run
nvidia-smi -L in ubuntu:24.04 containers. They test Job and Deployment DRA integration.
test_gpu_dynmig.bats -- 0/5 (entire file blocked)
#
Test
Tags
Compatible
Why
21
DynMIG: inspect device attributes in resource slice (gpu)
fastfeedback, dynmig, version-specific
NO*
setup_file blocks
22
DynMIG: inspect device attributes in resource slice (mig)
fastfeedback, dynmig, version-specific
NO
Expects MIG devices
23
DynMIG: 1 pod, 1 MIG
fastfeedback, dynmig
NO
MIG not supported
24
DynMIG: 1 pod, 2 containers (1 MIG each)
fastfeedback, dynmig
NO
MIG not supported
25
DynMIG: 1 pod, 1 MIG + TimeSlicing config
fastfeedback, dynmig
NO
MIG not supported
*Test #21 would work in isolation (just reads ResourceSlice gpu attributes) but the
file's setup_file hook reinstalls the driver with DynamicMIG=true and waits for
"About to announce device gpu-0-mig-1g" in logs. This never appears with mock NVML,
so setup_file itself times out before any test runs.
test_basics.bats -- 9 tests, not suitable for mock e2e
#
Test
Compatible
Why
27
VERSION targets
YES (standalone)
Pure Makefile, no K8s
28
confirm no kubelet plugin pods
NO
Expects clean state (no driver)
29
GPU Operator installed
NO
We don't have GPU Operator
30
helm install
YES (if run first)
But we already installed
31
helm list validate
YES
Metadata check
32
get CRD computedomains
YES
CRD existence
33
wait for plugin & controller READY
YES
Pod readiness
34
validate controller image spec
YES
Pod spec check
35
SIGUSR2 handler
YES
Signal handling
These tests assume a specific lifecycle: clean cluster -> install -> validate. They'd
conflict with our already-deployed driver. The Lambda CI script also skips test_basics.bats
in its tests-gpu-single target for the same reason (no GPU Operator in Lambda).
Recommended Test Plan
Tier 1 -- High Confidence (5 tests)
Run first. These are the simplest DRA allocation tests.
test_gpu_basic.bats
Filter: !version-specific
Tier 2 -- DRA Lifecycle and Robustness (7 tests)
Core DRA mechanics: allocation, release, reallocation, CEL selectors, metrics.
test_gpu_robustness.bats
Filter: skip device-count test (#14) unless /dev/nvidia* symlinks are created.
Tier 3 -- Scheduling Features (2 tests)
Extended resource bridge (k8s 1.35+) and TimeSlicing.
test_gpu_extres.bats
test_gpu_sharing.bats
Filter: !gpu-sharing to skip MPS test, or try it and see.
Tier 4 -- Job/Deployment Allocation (2 tests)
Job completions and Deployment replicas with DRA.
test_gpu_cuda_workloads.bats
Filter: !gpu-busgrind (busGrind is the only test with a unique tag; the CUDA demo
suite test has tag gpu-workloads shared by Job/Deployment tests, so it cannot be
individually filtered by tag -- would need to be skipped by test name).
Device nodes at wrong path. Tests expect /dev/nvidia[0-9]* on the host.
Fixed:setup-mock-gpu.sh creates device nodes at both ${DEV_ROOT}/ and
${DRIVER_ROOT}/dev/ (the latter is needed for CDI discovery via getDevRoot()).
CDI spec generation needed three k8s-test-infra fixes (PR #306):
a. IsMigDeviceHandle returning NOT_SUPPORTED -- implemented to return false
b. No SONAME in mock .so -- added -Wl,-soname,libnvidia-ml.so.1 to Makefile
c. Non-hex UUIDs rejected by nvidia-container-toolkit -- fixed all profiles
Result: CDI specs generated correctly, workload pods start 1/1 Running,
nvidia-smi executes inside them.
CDI spec file permissions.sudo tee created the file, then cat >>
appended without sudo. Fixed: all appends use sudo tee -a.
SKIP_CLEANUP variable collision. The e2e script exported SKIP_CLEANUP=true
for the BATS Makefile, clobbering the EXIT trap's cleanup check.
Fixed: scoped to the make command, EXIT trap now also calls
cleanup-mock-gpu.sh.
TEST_ALT_PROC_DEVICES never exported in e2e-test.sh.
Fixed: exported alongside other BATS env vars.
go install binaries not on PATH.common.sh now adds GOPATH/bin to PATH.
BATS runner needs --network host to reach Kind API server (127.0.0.1).
Not yet added to e2e-test.sh -- the BATS Makefile RUN_BATS macro does not
include it. Manual test runs required --network host passed explicitly.
Kind image cache stale tag. When rebuilding with same tag, kind load may
not replace the image. Use ctr images rm first, or fresh Kind cluster.
setup_suite expects clean helm state. Uninstall before running BATS.
Resolved: Device Visibility + Deterministic UUIDs
Two fixes in k8s-test-infra implement-cdi-nvml-functions branch:
Device visibility filtering (engine.go:detectVisibleDevices): During
nvmlInit(), scan /dev/nvidia<N> nodes. Only expose devices whose nodes exist.
CDI injects only the allocated GPU's device node, so workload pods see the correct
count. No filtering when all or no nodes exist (host/driver-pod context).
Deterministic UUIDs (engine.go:createDefaultDevices): When no YAML config
is found (workload pods), use GPU-00000000-0000-0000-0000-00000000000N instead
of random dgxa100 UUIDs. Two pods sharing the same GPU see the same UUID.
Actual Test Run Results (2026-04-15)
test_gpu_basic.bats -- ATTEMPTED
Result: Pod started 1/1 Running, nvidia-smi executed, but test FAILED.
not ok 1 GPUs: 1 pod(s), 1 full GPU
The pod got a GPU allocated (allocated,reserved), CDI injected device nodes
and nvidia-smi+library, the pod ran nvidia-smi successfully. But mock nvidia-smi
showed all 8 GPUs instead of just the 1 allocated GPU. The test checks:
Root cause: Mock NVML nvmlDeviceGetCount() always returns all configured
devices. It doesn't honor NVIDIA_VISIBLE_DEVICES env var for per-container
GPU filtering. Real NVIDIA drivers filter visible devices based on this env var
(set by CDI) or cgroup device restrictions.
Final Results (2026-04-16)
38/38 ok (17 pass, 21 skip) in GHA CI across 13 BATS files, 21m30s.
Mock NVML for GB200 Emulation: Technical Deep-Dive and Experiment Plan
1. How Mock NVML Actually Works (Peeling the Onion)
The mock NVML in github.com/nvidia/k8s-test-infra is a drop-in replacement for
NVIDIA's libnvidia-ml.so shared library. It compiles Go code into a C shared object
that exports the same symbols nvidia-smi (and any NVML consumer) calls via dlopen/dlsym.
There is no kernel driver, no GPU hardware, no CUDA runtime underneath. It's turtles all
the way down to a YAML file.
Layer 1: The C ABI Surface (Bridge Layer)
Location:pkg/gpu/mocknvml/bridge/
The bridge is a set of Go files that use //export CGo directives to produce C-callable
functions. When built with go build -buildmode=c-shared, Go emits a .so with a proper
ELF symbol table containing every exported function.
There are ~110 hand-implemented NVML functions covering:
An additional ~287 stub functions are auto-generated in stubs_generated.go. Each stub
returns NVML_ERROR_NOT_SUPPORTED. This ensures that even when nvidia-smi (or another NVML
consumer) probes for a function that is rarely used, it gets a clean error instead of a
missing symbol crash.
Version aliasing is critical. NVML headers define macros like
#define nvmlDeviceGetPciInfo nvmlDeviceGetPciInfo_v3. nvidia-smi was compiled against
these headers, so it dlsym()s the unversioned name. The Makefile uses ELF linker
aliases (--defsym) to map unversioned to versioned:
When nvmlInit() is called, the bridge calls engine.GetEngine().Init(). The engine is a
Go singleton (protected by sync.Once) that:
Loads configuration (YAML file or environment variables)
Creates a dgxa100.Server from go-nvml/pkg/nvml/mock/dgxa100 -- this is NVIDIA's
own mock A100 implementation with sensible defaults for every NVML field
Wraps each device in a ConfigurableDevice that overlays YAML-driven property
values on top of the dgxa100 base device
Registers C-safe handles in a HandleTable that maps opaque uintptr handles
to Go device objects (so C callers never hold raw Go pointers)
The configuration resolution order is:
1. MOCK_NVML_CONFIG env var → explicit YAML path
2. /proc/self/maps auto-discovery → reads the loaded .so's filesystem path,
navigates to <driver_root>/config/config.yaml
3. MOCK_NVML_NUM_DEVICES env var → number of devices (default: 8)
MOCK_NVML_DRIVER_VERSION env var → driver version string (default: 550.163.01)
4. Built-in defaults → 8x A100, driver 550.163.01
The /proc/self/maps trick is elegant: when libnvidia-ml.so is loaded into a process,
its path appears in /proc/self/maps. The engine reads this file, finds the line containing
libnvidia-ml.so, extracts the filesystem path, walks up from <root>/usr/lib64/ to
<root>/, and looks for config/config.yaml at that root. This means if you install the
library at /var/lib/nvml-mock/driver/usr/lib64/libnvidia-ml.so.570.170.01, it will
automatically find /var/lib/nvml-mock/driver/config/config.yaml -- no environment
variables needed.
Each ConfigurableDevice wraps a dgxa100.Device and overrides its methods based on YAML
configuration. The YAML structure has two sections:
device_defaults: Properties applied to every GPU (name, architecture, memory, power, etc.)
devices[]: Per-index overrides (UUID, PCI bus ID, serial, minor number, Grace CPU pairing)
The merge logic in config.go:mergeDeviceOverride() copies non-zero override values into
the defaults. This means you can define 8 GPUs that share all properties except UUID, PCI
bus ID, and Grace CPU pair.
Layer 4: The Build Artifact
Location:pkg/gpu/mocknvml/Makefile
CGO_LDFLAGS_ALLOW='--defsym' CGO_LDFLAGS='$(VERSION_ALIASES)' \
go build -buildmode=c-shared -o libnvidia-ml.so.550.163.01 ./bridge
This produces a standard ELF shared library. The symlink chain
libnvidia-ml.so → libnvidia-ml.so.1 → libnvidia-ml.so.550.163.01 matches what a real
NVIDIA driver installation creates, so ldconfig, LD_LIBRARY_PATH, and RPATH all work
normally.
For cross-compilation on macOS, make docker-build runs the build inside a
golang:1.25.0-bookworm container.
Key GB200-specific attributes exposed through NVML:
Attribute
Value
NVML Function
Name
"NVIDIA GB200 NVL"
nvmlDeviceGetName
Architecture
Blackwell
nvmlDeviceGetArchitecture
Compute Capability
10.0
nvmlDeviceGetCudaComputeCapability
CUDA Cores
18,432
(config field, not directly in NVML)
Memory
192 GiB HBM3e
nvmlDeviceGetMemoryInfo
BAR1
512 GiB
nvmlDeviceGetBAR1MemoryInfo
Memory Bus Width
5120-bit (dgxa100 default)
nvmlDeviceGetMemoryBusWidth
Power Limit
1000W (max 1200W)
nvmlDeviceGetPowerManagementLimit
PCIe
Gen6 x16
nvmlDeviceGetMaxPcieLinkGeneration
NVLink
v5, 18 links @ 100GB/s
nvmlDeviceGetNvLinkState
NVLink-C2C
Enabled (to Grace CPU)
Via NVLink remote device type = "cpu"
ECC
Enabled
nvmlDeviceGetEccMode
Driver
570.170.01 (patched by setup script)
nvmlSystemGetDriverVersion
CUDA
12.6
nvmlSystemGetCudaDriverVersion_v2
The features section captures Blackwell-specific capabilities:
2nd gen Transformer Engine, FP4/FP8, confidential compute, decompression engine, 5th gen tensor cores
The grace_superchip section captures the CPU side:
72 Grace cores, 480GB LPDDR5X, coherent memory via NVLink-C2C
3. How nvidia-smi Works Without a GPU
The Docker build (deployments/nvml-mock/Dockerfile) extracts the real nvidia-smi ELF
binary from the nvidia-utils-550 Ubuntu package. This is the same binary that ships
with real NVIDIA drivers. It is then patched with patchelf:
nvidia-smi is not a kernel-space tool. It is a userspace binary that:
dlopen("libnvidia-ml.so.1") -- finds the library via RPATH/LD_LIBRARY_PATH
dlsym() each NVML function it needs (using unversioned names)
Calls nvmlInit(), then queries device properties, formats output, exits
Because our mock libnvidia-ml.so.1 exports all the same symbols and returns plausible
values, nvidia-smi runs normally and displays the fake GPU information. It never touches
/dev/nvidia* devices directly for query operations -- those devices only matter for CUDA
compute and container runtime injection.
4. Host-to-Container Injection: CDI and Device Nodes
The setup.sh script creates the full mock driver environment on the host filesystem:
A CDI (Container Device Interface) spec is generated at /var/run/cdi/nvidia.yaml.
This is the modern replacement for the legacy nvidia-container-runtime hook. CDI tells
the container runtime (containerd, CRI-O) to bind-mount specific host paths into the
container when a workload requests nvidia.com/gpu:
Bind-mount libnvidia-ml.so.1 into /usr/lib64/
Bind-mount nvidia-smi into /usr/bin/
Bind-mount /dev/nvidia{N} device nodes
Run nvidia-cdi-hook update-ldcache to rebuild the container's ld.so.cache
The device nodes are fake -- they are mknod-created character devices with the correct
major/minor numbers (195 for nvidia, 510 for nvidia-uvm) but no kernel driver backing them.
They exist purely for path compatibility. NVML queries go through the userspace library,
not through device ioctls.
5. Experiment Plan: GB200 Emulation on a Brev CPU-Only Instance
5.1 Goal
Stand up a CPU-only Linux machine where:
nvidia-smi on the host shows 8x NVIDIA GB200 NVL GPUs
nvidia-smi inside a Docker container shows the same
The NVIDIA Container Toolkit CDI integration works end-to-end
This validates readiness for Prow CI jobs that test DRA driver behavior against GB200
5.2 Prerequisites
A brev instance running Ubuntu 22.04 or Debian Bookworm (amd64). Needs to be beefy
enough to build Go code (4+ cores, 8GB+ RAM recommended for the CGo shared library build).
Root/sudo access (needed for mknod, Docker, library installation).
Docker installed and running.
Go 1.25+ installed (for building the mock library from source).
git to clone the repository.
5.3 Phase 1: Build the Mock NVML Library
# Clone the repo
git clone https://github.com/nvidia/k8s-test-infra.git
cd k8s-test-infra
# Option A: Build natively on Linux (requires Go 1.25+, gcc, make)cd pkg/gpu/mocknvml
make clean && make
# Produces: libnvidia-ml.so.550.163.01 + symlinks# Option B: Build in Docker (works from any host with Docker)cd pkg/gpu/mocknvml
make docker-build
# Same output, built inside golang:1.25.0-bookworm container
Checkpoint: You should see three files:
libnvidia-ml.so → libnvidia-ml.so.1
libnvidia-ml.so.1 → libnvidia-ml.so.550.163.01
libnvidia-ml.so.550.163.01 (the actual ELF shared library)
We'll replicate what setup.sh does, but manually, so we can verify each step.
# Create the driver root structure
DRIVER_ROOT=/var/lib/nvml-mock/driver
sudo mkdir -p $DRIVER_ROOT/usr/lib64
sudo mkdir -p $DRIVER_ROOT/usr/bin
sudo mkdir -p $DRIVER_ROOT/config
sudo mkdir -p /var/lib/nvml-mock/dev
# Copy the mock library with the GB200 driver version
DRIVER_VERSION=570.170.01
sudo cp libnvidia-ml.so.550.163.01 $DRIVER_ROOT/usr/lib64/libnvidia-ml.so.$DRIVER_VERSION
sudo ln -sf libnvidia-ml.so.$DRIVER_VERSION$DRIVER_ROOT/usr/lib64/libnvidia-ml.so.1
sudo ln -sf libnvidia-ml.so.1 $DRIVER_ROOT/usr/lib64/libnvidia-ml.so
# Copy the GB200 config
sudo cp ../configs/mock-nvml-config-gb200.yaml $DRIVER_ROOT/config/config.yaml
# Inject num_devices (use 8 for full GB200 NVL simulation)
sudo sed -i '/^system:/a\ num_devices: 8'$DRIVER_ROOT/config/config.yaml
Checkpoint: Verify library installation:
ls -la $DRIVER_ROOT/usr/lib64/libnvidia-ml.so*# Should show the symlink chain
readelf -d $DRIVER_ROOT/usr/lib64/libnvidia-ml.so.$DRIVER_VERSION| grep SONAME
# Should show libnvidia-ml.so.1 or similar
5.5 Phase 3: Get and Patch nvidia-smi
# Download nvidia-smi from the NVIDIA Ubuntu repo# (Same approach as the Dockerfile)
sudo apt-get update
sudo apt-get install -y patchelf
# Add NVIDIA repo and download nvidia-utils
curl -fsSL "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub"| \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia.gpg
echo"deb [signed-by=/usr/share/keyrings/nvidia.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 /"| \
sudo tee /etc/apt/sources.list.d/nvidia.list
sudo apt-get update
# Download (don't install) the nvidia-utils package to extract nvidia-smicd /tmp
apt-get download nvidia-utils-550
dpkg-deb -x nvidia-utils-550*.deb /tmp/nvidia-utils
sudo cp /tmp/nvidia-utils/usr/bin/nvidia-smi $DRIVER_ROOT/usr/bin/nvidia-smi
sudo chmod +x $DRIVER_ROOT/usr/bin/nvidia-smi
# Patch RPATH so nvidia-smi finds our mock library at ../lib64 relative to itself
sudo patchelf --set-rpath '$ORIGIN/../lib64'$DRIVER_ROOT/usr/bin/nvidia-smi
Checkpoint: Verify RPATH is set:
patchelf --print-rpath $DRIVER_ROOT/usr/bin/nvidia-smi
# Expected: $ORIGIN/../lib64
ldd $DRIVER_ROOT/usr/bin/nvidia-smi
# libnvidia-ml.so.1 should resolve to the mock library path
5.6 Phase 4: Test nvidia-smi on the Host
# Run nvidia-smi against the mock library$DRIVER_ROOT/usr/bin/nvidia-smi
If the auto-discovery via /proc/self/maps doesn't work inside the container (path
mismatch), fall back to the explicit env var. A cleaner approach preserving the
<root>/usr/lib64 ← → <root>/config/ layout:
The NVIDIA Container Toolkit in CDI mode does not require kernel modules or a real GPU.
It reads CDI specs from /etc/cdi/ or /var/run/cdi/ and instructs the container runtime
to inject the specified mounts and device nodes.
nvidia-ctk cdi list
# Expected: nvidia.com/gpu=0, nvidia.com/gpu=1, ..., nvidia.com/gpu=all
5.11 Phase 9: Test nvidia-smi via CDI-Injected Container
# Run a container requesting all GPUs via CDI
docker run --rm \
--device nvidia.com/gpu=all \
-e MOCK_NVML_CONFIG=/run/nvidia/driver/config/config.yaml \
debian:bookworm-slim \
nvidia-smi
Note on config discovery inside the container: The CDI spec mounts the library at
/usr/lib64/libnvidia-ml.so.1. The auto-discovery via /proc/self/maps will see this
path, navigate to /usr/lib64/../../config/config.yaml = /config/config.yaml, which
won't exist. Two options:
Add a config mount to the CDI spec (add another mount entry mapping config.yaml)
Use the MOCK_NVML_CONFIG env var (add to CDI spec env section, or pass via -e)
For the experiment, option 2 with an additional bind mount is simplest:
Expected: Full nvidia-smi output showing 8x NVIDIA GB200 NVL GPUs inside the container.
6. Key Considerations for Prow CI
What This Enables
DRA driver's kubelet-plugin calls NVML to discover GPUs and publish ResourceSlices.
With mock NVML, it will discover 8x GB200 NVL GPUs and publish them correctly.
Scheduling, allocation, and claim lifecycle can be tested end-to-end.
GPU feature discovery (GFD) will report Blackwell architecture, compute capability 10.0,
192 GiB memory, etc.
Both gpus and compute-domains containers run healthy on CPU-only nodes with
two small DRA driver changes (see "IMEX Channel Mocking" below).
What This Does NOT Enable
No actual CUDA compute. cudaMalloc/cudaLaunchKernel from the mock CUDA library are
stubs -- they allocate host memory and no-op kernels.
MIG device enumeration returns NOT_FOUND (no MIG partitioning support in mock).
No real NVLink bandwidth or topology -- the values are static from config.
No GPU process tracking -- nvmlDeviceGetComputeRunningProcesses returns empty.
No real IMEX channel allocation -- channels are enumerated but not functional.
Mock NVML Version vs nvidia-smi Version
The mock library's build version (currently 550.163.01 in the Makefile) and the nvidia-smi
binary version (from nvidia-utils-550) must be compatible. nvidia-smi 550 will work with a
mock library that reports driver version 570.170.01 (via the YAML config) because the version
displayed comes from the NVML API return value, not from the ELF soname. However, if
nvidia-smi probes for functions that only exist in 570+ and the mock returns NOT_SUPPORTED,
some query modes might show "N/A" for those fields.
IMEX Channel Mocking for Compute Domains
The compute-domain-kubelet-plugin container reads /proc/devices to find the
nvidia-caps-imex-channels device major number and unmounts /proc/driver/nvidia. On
CPU-only nodes neither exists. Two changes in the DRA driver enable this:
internal/common/nvcaps.go: procDevicesPath reads from ALT_PROC_DEVICES_PATH
env var (falls back to /proc/devices). UsingAltProcDevices() helper signals mock mode.
cmd/compute-domain-kubelet-plugin/nvlib.go: Skips /proc/driver/nvidia unmount
when UsingAltProcDevices() returns true.
The Helm chart exposes altProcDevices which mounts a host file at /alt-proc-devices and
sets the env var. setup-mock-gpu.sh generates the fake /proc/devices content with
nvidia-caps-imex-channels (major 235) and nvidia-caps (major 236) entries.
The mock driver version must be >= 570.158.01 to satisfy the IMEXDaemonsWithDNSNames
feature gate. setup-mock-gpu.sh defaults to 570.170.01 for Blackwell profiles.
No feature gate overrides needed. Both containers reach 2/2 Running.
Why We Don't Use the k8s-test-infra nvml-mock Helm Chart
The nvml-mock Helm chart at deployments/nvml-mock/helm/nvml-mock/ in k8s-test-infra
is designed for its own DaemonSet-based deployment pattern (device plugin, GPU Operator
validation). It does NOT handle the DRA driver's specific needs:
No device nodes inside driver/dev/ (needed for CDI getDevRoot() discovery)
No mock /proc/devices with nvidia-caps-imex-channels (needed for compute-domains)
No mock fabric-imex-mgmt capability file
No driver version override to 570.x (needed for IMEXDaemonsWithDNSNames gate)
Our hack/ci/mock-nvml/setup-mock-gpu.sh handles all of these. The Prow job uses:
setup-mock-gpu.sh to install mock NVML on the host (builds library, device nodes,
CDI spec, IMEX mocks, driver version patching)
These changes to the mock NVML library were needed for CDI spec generation
(all merged via NVIDIA/k8s-test-infra PR #306):
IsMigDeviceHandle: Must return SUCCESS with isMigDevice=0. Without this,
nvidia-container-toolkit CDI spec generator fails with ERROR_NOT_SUPPORTED.
SONAME in ELF header: Add -Wl,-soname,libnvidia-ml.so.1 to Makefile. Without
the SONAME, the CDI hook only creates .so symlink, not .so.1 which nvidia-smi
dlopen's.
Valid hex UUIDs: All GPU UUIDs must be valid hex format
GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Non-hex prefixes like GPU-GB200- are
rejected by nvidia-container-toolkit.
README example UUID: Helm chart README also updated to use hex UUIDs.
Device Node Placement
Mock device nodes (mknod) must be created at <driver_root>/dev/nvidia* (inside the
driver root), not as a sibling directory. The DRA driver's getDevRoot() checks for
<driverRoot>/dev/ directory existence to set the CDI device discovery root. If the
devices are only at /var/lib/nvml-mock/dev/ (sibling), the CDI discoverer looks at
the pod's own /dev and finds nothing. setup-mock-gpu.sh now creates device nodes
at both locations.
CI Script Hardening
Several issues were identified and fixed in the CI scripts:
CDI spec file appends now use sudo tee -a (not bare cat >> on root-owned file)
SKIP_CLEANUP for BATS is scoped to the make command (doesn't clobber EXIT trap)
EXIT trap calls cleanup-mock-gpu.sh for host artifact removal
TEST_ALT_PROC_DEVICES exported in e2e-test.sh for BATS reinstalls
GOPATH/bin added to PATH in common.sh (for go install'd binaries)
Current Status (2026-04-15)
Working end-to-end:
Mock NVML build with SONAME + IsMigDeviceHandle + hex UUIDs
38/38 ok (17 pass, 21 skip) in GitHub Actions CI (2026-04-16, PR #1043, 21m30s).
13 BATS files across GPU + CD test suites. Tests skip via MOCK_NVML=true env var.
21 skipping: CUDA demo + busGrind (2), MPS (1), updowngrade (1),
CD IMEX/misc/logging (10), CD failover (4), CD mnnvl (2), CD updowngrade (1).
Enabled by two additional mock NVML fixes (also in PR #306):
Device visibility filtering: nvmlInit() scans /dev/nvidia<N> nodes and
filters DeviceGetCount/DeviceGetHandleByIndex to only expose devices whose
nodes exist. CDI injects only the allocated device node, so workload pods see
the correct GPU count.
Deterministic UUIDs: Default (no-config) path uses index-based UUIDs
(GPU-00000000-0000-0000-0000-00000000000N) so pods sharing a GPU see identical
identifiers.
Ran first BATS test. Pod started and nvidia-smi -L ran, but test failed because
mock nvidia-smi shows ALL 8 GPUs instead of just the 1 allocated GPU. The mock NVML
library doesn't honor NVIDIA_VISIBLE_DEVICES or cgroup-based GPU filtering.
not ok 1 GPUs: 1 pod(s), 1 full GPU -- pod started (1/1 Running) but
nvidia-smi showed 8 GPUs instead of 1 (wc -l check failed)
This is the last remaining blocker for BATS tests passing.
(Shows A100 because mock NVML config auto-discovery falls back to dgxa100 defaults
when the driver pod's /proc/self/maps path doesn't match the config layout.
The DRA ResourceSlice correctly shows GB200 NVL attributes from the config.)
BATS test_gpu_basic.bats results:
1..4
not ok 1 GPUs: 1 pod(s), 1 full GPU in 2157ms
not ok 2 GPUs: 2 pod(s), 1 full GPU each in 2101ms
not ok 3 GPUs: 2 pod(s), 1 full GPU (shared, 1 RC) in 907ms
not ok 4 GPUs: 1 pod(s), 2 cntrs, 1 full GPU (shared, 1 RCT) in 860ms
All 4 tests: pods started successfully, nvidia-smi ran, but GPU count check
failed (mock shows all 8 GPUs, tests expect 1 or 2).
Added device visibility filtering to mock NVML: during nvmlInit(), scan for
/dev/nvidia<N> device nodes and only expose devices whose nodes exist. CDI injects
only the allocated GPU's device node, so workload pods now see the correct GPU count.
BATS test_gpu_basic.bats results:
1..4
ok 1 GPUs: 1 pod(s), 1 full GPU in 3788ms
ok 2 GPUs: 2 pod(s), 1 full GPU each in 5038ms
not ok 3 GPUs: 2 pod(s), 1 full GPU (shared, 1 RC) in 2577ms
not ok 4 GPUs: 1 pod(s), 2 cntrs, 1 full GPU (shared, 1 RCT) in 860ms
Tests 1-2 PASS. Device count filtering works correctly -- pods see only
their allocated GPUs.
Tests 3-4 FAIL on UUID mismatch (not GPU count). Two pods sharing the same
GPU via a single ResourceClaim see different UUIDs because the mock NVML config
auto-discovery via /proc/self/maps doesn't work inside workload pods (falls
back to dgxa100 random UUIDs instead of configured GPU-b200b200-... UUIDs).
The visibility filtering itself is correct -- both pods see exactly 1 GPU.
Scenario 7: Deterministic UUIDs -- All 4 Tests Pass (2026-04-15)
Fixed the UUID mismatch by using deterministic UUIDs (GPU-00000000-0000-0000-0000-00000000000N)
in the default (no-config) device creation path. Two pods sharing the same GPU now
see the same UUID because the UUID is derived from the device index, not randomly generated.
BATS test_gpu_basic.bats results (brev):
ok 1 GPUs: 1 pod(s), 1 full GPU in 9777ms
ok 2 GPUs: 2 pod(s), 1 full GPU each in 5067ms
ok 3 GPUs: 2 pod(s), 1 full GPU (shared, 1 RC) in 4008ms
ok 4 GPUs: 1 pod(s), 2 cntrs, 1 full GPU (shared, 1 RCT) in 4940ms
Scenario 8: GitHub Actions CI -- Final State (2026-04-16)
38/38 ok (17 pass, 21 skip) across 13 BATS files on ubuntu-latest, 21m30s:
ok 1 GPUs: 1 pod(s), 1 full GPU
ok 2 GPUs: 2 pod(s), 1 full GPU each
ok 3 GPUs: 2 pod(s), 1 full GPU (shared, 1 RC)
ok 4 GPUs: 1 pod(s), 2 cntrs, 1 full GPU (shared, 1 RCT)
ok 5 GPUs: single GPU runs CUDA demo suite # skip requires real CUDA compute
ok 6 GPUs: Job with ResourceClaimTemplate allocates GPUs to completions
ok 7 GPUs: Deployment with 2 replicas gets distinct GPUs simultaneously
ok 8 GPUs: TimeSlicing — 2 containers share GPU with Short interval
ok 9 GPUs: MPS — 2 containers share GPU with MPS config # skip requires real MPS daemon
ok 10 GPUs: kubelet-plugin exposes Prometheus metrics
ok 11 GPUs: ResourceClaim lifecycle
ok 12 GPUs: GPU re-acquired after pod delete and re-create
ok 13 GPUs: pod with two ResourceClaimTemplates gets two distinct GPUs
ok 14 GPUs: CEL selector matches GPU by architecture attribute
ok 15 GPUs: CEL selector rejects non-matching productName
ok 16 GPUs: ResourceSlice device count matches host GPU count
ok 17 GPUs: rapid claim create/delete does not leak resources
ok 18 GPUs: handle legacy 'nvidia.com/gpu: 1' (with DRAExtendedResource)
ok 19 GPUs: shared RC across 20 pods, 1 repetitions (stress)
ok 20 GPUs: upgrade: wipe-state, install-last-stable, upgrade # skip requires last-stable image
ok 21-30 CDs: IMEX/misc/logging (10 tests) # skip requires IMEX daemon
ok 31-34 CDs: failover (4 tests) # skip requires multi-node NVLink
ok 35-36 CDs: mnnvl workload (2 tests) # skip requires multi-node NCCL
ok 37-38 CDs: updowngrade (2 tests) # skip requires IMEX daemon
17 passing: 4 basic + 2 job/deployment + 1 timeslicing + 8 robustness + 1 extres + 1 stress
21 skipping: 2 CUDA + 1 MPS + 1 updowngrade + 10 CD IMEX + 4 CD failover + 2 CD mnnvl + 2 CD updowngrade (1 overlaps with GPU updowngrade -- actual total is correct at 21)
