Simulating GPU Hardware Failure on OpenShift with NVIDIA GPU Operator

gpu-failure-simulation.md

Simulating GPU Hardware Failure on OpenShift with NVIDIA GPU Operator

TL;DR

When a GPU fails in a way that does not produce a critical XID error in the kernel (e.g., XIDs 48, 74, 79, 94, 95), the NVIDIA device plugin does not detect the failure. The node continues advertising nvidia.com/gpu: 1, no taint or cordon is applied, and the scheduler keeps sending GPU pods to the broken node. Those pods start, fail with CUDA errors (invalid device ordinal), and exit — but the node remains eligible for more GPU workloads. This document reproduces the issue end-to-end on an OpenShift 4.21 cluster on AWS using a g4dn.xlarge instance (NVIDIA Tesla T4).

The failure was simulated by removing the GPU from the PCI bus:

echo 1 > /sys/bus/pci/devices/0000:00:1e.0/remove

The kernel logged:

NVRM: Attempting to remove device 0000:00:1e.0 with non-zero usage count!

And nvidia-smi from the driver container confirmed the GPU was gone:

No devices were found

Despite this, the node kept advertising nvidia.com/gpu: 1 with no taints applied.

Prerequisites

• OpenShift 4.x cluster on AWS
• At least one worker node with an NVIDIA GPU (this guide uses g4dn.xlarge with a single Tesla T4)
• oc CLI authenticated as system:admin (or equivalent)
• Access to OperatorHub (Red Hat and Certified catalogs)

install-config.yaml

The cluster was deployed with the following install-config.yaml. The key part is the worker pool using g4dn.xlarge:

additionalTrustBundlePolicy: Proxyonly
apiVersion: v1
baseDomain: example.com
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    aws:
      type: g4dn.xlarge
  replicas: 1
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 1
metadata:
  name: testcluster
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-west-2
publish: External
pullSecret: '<your-pull-secret>'
sshKey: '<your-ssh-key>'

Step 1: Install the NFD Operator

The GPU Operator requires Node Feature Discovery (NFD) to label GPU nodes. Install NFD first.

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-nfd
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: nfd-og
  namespace: openshift-nfd
spec:
  targetNamespaces:
    - openshift-nfd
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: nfd
  namespace: openshift-nfd
spec:
  channel: stable
  name: nfd
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic

Apply and wait for the operator CSV to reach Succeeded:

oc apply -f nfd-subscription.yaml

# Wait for the operator
oc get csv -n openshift-nfd -w

Create the NFD instance

apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
  name: nfd-instance
  namespace: openshift-nfd
spec:
  operand:
    servicePort: 12000
  workerConfig:
    configData: |
      core:
        sleepInterval: 60s
      sources:
        pci:
          deviceClassWhitelist:
            - "0200"
            - "03"
            - "12"
          deviceLabelFields:
            - "vendor"

oc apply -f nfd-instance.yaml

Verify the GPU node is labeled (NVIDIA PCI vendor ID is 10de):

oc get nodes -l feature.node.kubernetes.io/pci-10de.present=true

Expected output:

NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-17-19.us-west-2.compute.internal    Ready    worker   19m   v1.34.6

Step 2: Install the NVIDIA GPU Operator

apiVersion: v1
kind: Namespace
metadata:
  name: nvidia-gpu-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: gpu-operator-og
  namespace: nvidia-gpu-operator
spec:
  targetNamespaces:
    - nvidia-gpu-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: gpu-operator-certified
  namespace: nvidia-gpu-operator
spec:
  channel: v26.3
  name: gpu-operator-certified
  source: certified-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic

oc apply -f gpu-operator-subscription.yaml

# Wait for the operator
oc get csv -n nvidia-gpu-operator -w

Create the ClusterPolicy

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  operator:
    defaultRuntime: crio
    use_ocp_driver_toolkit: true
  daemonsets: {}
  driver:
    enabled: true
    upgradePolicy:
      autoUpgrade: true
      maxParallelUpgrades: 1
      maxUnavailable: "25%"
  devicePlugin:
    enabled: true
  dcgm:
    enabled: true
  dcgmExporter:
    enabled: true
  gfd:
    enabled: true
  migManager:
    enabled: false
  nodeStatusExporter:
    enabled: true
  toolkit:
    enabled: true
  validator:
    plugin:
      env:
        - name: WITH_WORKLOAD
          value: "false"

oc apply -f cluster-policy.yaml

# Wait for the ClusterPolicy to reach "ready" (can take ~8 minutes for driver compilation)
watch oc get clusterpolicy gpu-cluster-policy -o jsonpath='{.status.state}'

Verify all GPU operator pods are running:

oc get pods -n nvidia-gpu-operator

Expected output:

gpu-feature-discovery-...                    1/1   Running     0
gpu-operator-...                             1/1   Running     0
nvidia-container-toolkit-daemonset-...       1/1   Running     0
nvidia-cuda-validator-...                    0/1   Completed   0
nvidia-dcgm-...                              1/1   Running     0
nvidia-dcgm-exporter-...                     1/1   Running     0
nvidia-device-plugin-daemonset-...           1/1   Running     0
nvidia-driver-daemonset-...                  2/2   Running     0
nvidia-node-status-exporter-...              1/1   Running     0
nvidia-operator-validator-...                1/1   Running     0

Verify the GPU is advertised:

oc get node -l feature.node.kubernetes.io/pci-10de.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}nvidia.com/gpu: {.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

Expected: nvidia.com/gpu: 1

Step 3: Validate the GPU Stack

Run a CUDA workload to confirm the GPU is functional:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-burn-test
  namespace: default
spec:
  restartPolicy: Never
  containers:
  - name: gpu-burn
    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
    resources:
      limits:
        nvidia.com/gpu: 1

oc apply -f gpu-test.yaml
oc wait --for=condition=Ready pod/gpu-burn-test --timeout=120s || true
oc logs gpu-burn-test

Expected output:

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Clean up:

oc delete pod gpu-burn-test

Step 4: Simulate GPU Failure

Find the GPU PCI address

oc debug node/<gpu-node> -- chroot /host lspci | grep -i nvidia

Output:

00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)

The PCI address is 0000:00:1e.0.

Remove the GPU from the PCI bus

This simulates a hardware failure where the GPU "falls off the bus":

oc debug node/<gpu-node> -- chroot /host bash -c '
echo 1 > /sys/bus/pci/devices/0000:00:1e.0/remove
'

Note: The write to remove may block or be rejected with NVRM: Attempting to remove device 0000:00:1e.0 with non-zero usage count! because the NVIDIA driver holds references to the device (device plugin, DCGM, persistenced all have the GPU open). Even when the kernel rejects the clean removal, the driver can end up in an inconsistent state where nvidia-smi reports "No devices were found" while the PCI device still appears in lspci.

Verify the GPU is broken

From the driver container:

DRIVER_POD=$(oc get pods -n nvidia-gpu-operator -o name | grep nvidia-driver-daemonset)
oc exec -n nvidia-gpu-operator $DRIVER_POD -c nvidia-driver-ctr -- nvidia-smi

Expected output:

No devices were found

Check dmesg

oc debug node/<gpu-node> -- chroot /host dmesg | grep -iE "NVRM|xid|nvidia.*error|GPU"

Observed output:

[ 5524.708906] NVRM: Attempting to remove device 0000:00:1e.0 with non-zero usage count!

Note: No XID error was emitted. This is the crux of the issue.

Why DCGM injection does not work

You might try injecting an XID via DCGM:

DCGM_POD=$(oc get pods -n nvidia-gpu-operator -l app=nvidia-dcgm -o jsonpath='{.items[0].metadata.name}')
oc exec -n nvidia-gpu-operator $DCGM_POD -- dcgmi test --inject --gpuid 0 -f 230 -v 79
# Successfully injected field info.

This writes a fake XID 79 into DCGM's in-memory field database, but the device plugin uses NVML event sets (kernel-level) for health checks, not DCGM field queries. The injection never reaches the NVML layer, so the device plugin remains unaware.

Step 5: Observe the Gap

After the GPU failure, check the cluster state:

# Node conditions
oc get node <gpu-node> -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\n"}{end}'

# Taints
oc describe node <gpu-node> | grep -A3 "Taints:"

# GPU resource counts
oc get node <gpu-node> -o jsonpath='Allocatable: {.status.allocatable.nvidia\.com/gpu}  Capacity: {.status.capacity.nvidia\.com/gpu}'

# Device plugin logs (look for health events)
oc logs -n nvidia-gpu-operator -l app=nvidia-device-plugin-daemonset --tail=20

Check	Expected (healthy behavior)	Actual
nvidia-smi	Reports GPU	"No devices found" (exit 6)
Node condition	Ready: True	Ready: True
Node taints	Taint applied (e.g., nvidia.com/gpu=unhealthy:NoSchedule)	No taints
nvidia.com/gpu Allocatable	0	1
nvidia.com/gpu Capacity	0	1
Device plugin logs	Health event logged, GPU marked unhealthy	Silent — no new log lines
dmesg XID	XID 79 (GPU has fallen off the bus)	No XID emitted

Step 6: Schedule a GPU Pod on the Broken Node

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test-after-failure
  namespace: default
spec:
  restartPolicy: Never
  containers:
  - name: gpu-test
    image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
    resources:
      limits:
        nvidia.com/gpu: 1

oc apply -f gpu-test-after-failure.yaml
oc wait --for=jsonpath='{.status.phase}'=Failed pod/gpu-test-after-failure --timeout=120s || true
oc logs gpu-test-after-failure

Output:

Failed to allocate device vector A (error code invalid device ordinal)!
[Vector addition of 50000 elements]

The pod was:

1. Scheduled to the broken GPU node (scheduler saw nvidia.com/gpu: 1 allocatable)
2. Allocated a GPU device by the device plugin
3. Failed at runtime because the GPU is non-functional

Findings

Root Cause

The NVIDIA k8s-device-plugin relies on NVML event sets to detect GPU health issues. It listens for specific critical XID errors (48, 74, 79, 94, 95) via the kernel driver's event channel. Certain GPU failure modes — including PCI-level device loss while the driver has open references — do not produce XID errors in dmesg. When no XID is emitted, the device plugin's health monitor never fires, and:

• The GPU remains in the device plugin's "healthy" list
• nvidia.com/gpu allocatable/capacity counts are not updated
• No node taint or cordon is applied
• The scheduler continues to treat the node as a valid GPU target

What the GPU Operator does NOT do automatically

• Cordon or drain the node on GPU failure
• Apply a taint to block GPU pod scheduling
• Withdraw the nvidia.com/gpu allocatable count
• Surface a node-level condition for GPU health

XID Reference

The device plugin ignores the following XIDs (non-critical):

XID	Description
13	Graphics engine exception
31	GPU memory page fault
43	GPU stopped processing
45	Preemptive cleanup
68	Video processor exception
109	Context switch timeout

Critical XIDs that should trigger health marking (but require the kernel to actually emit them):

XID	Description
48	Double-bit ECC error (uncorrectable)
74	GPU is being decommissioned
79	GPU has fallen off the bus
94	Contained ECC error
95	Uncontained ECC error

Recommendations

1. The GPU Operator / device plugin should implement an active health probe (e.g., periodic nvmlDeviceGetHandleByIndex or equivalent) rather than relying solely on passive XID event listening.
2. A node taint (e.g., nvidia.com/gpu=unhealthy:NoSchedule) should be automatically applied when GPU health checks fail.
3. The nvidia.com/gpu allocatable count should be withdrawn when a GPU is detected as unhealthy.
4. More specific alerts should be surfaced (e.g., NodeHasUnreachableGPU), tied to both the node name and the specific failure, rather than only the generic GPUOperatorReconciliationFailed.

Component Versions

Component	Version
OpenShift	4.21.14
Kubernetes	v1.34.6
RHCOS kernel	5.14.0-570.112.1.el9_6.x86_64
NFD Operator	nfd.4.21.0-202604221819
GPU Operator	gpu-operator-certified.v26.3.1
NVIDIA driver	580.126.20 (Open Kernel Module)
NVIDIA device plugin	v0.19.0
DCGM exporter	4.5.1-4.8.0
GPU hardware	NVIDIA Tesla T4 (g4dn.xlarge)

Cleanup

Rescan the PCI bus to recover the GPU (may require a node reboot if the driver state is too corrupted):

oc debug node/<gpu-node> -- chroot /host bash -c 'echo 1 > /sys/bus/pci/rescan'

Delete test pods:

oc delete pod gpu-test-after-failure -n default --ignore-not-found