- An OpenShift cluster on AWS with UEFI Secure Boot enabled and NFD (Node Feature Discovery) deployed. See Enabling UEFI Secure Boot for OCP Workers on AWS for the setup procedure.
- GPU worker nodes (e.g.,
g4dn.xlarge) labeled by NFD withfeature.node.kubernetes.io/pci-10de.present=true.
Precompiled driver images are kernel-specific. First, determine the kernel version running on your RHCOS nodes:
oc get nodes -o wide
# Look for KERNEL-VERSION column, e.g.: 5.14.0-570.112.1.el9_6.x86_64Alternatively, extract it from the release image:
oc image info -a /path/to/pull-secret.json \
$(oc adm release info --image-for=rhel-coreos) 2>/dev/null | grep ostree.linux
# e.g.: ostree.linux=5.14.0-570.112.1.el9_6.x86_64Then list available tags for the precompiled driver image and find one matching your kernel:
skopeo list-tags docker://registry.stage.redhat.io/nvidia/gpu-driver-rhel9 | \
python3 -c "
import json, sys
data = json.load(sys.stdin)
tags = [t for t in data['Tags'] if not t.endswith('-source') and 'sha256' not in t]
# Filter for your kernel
kernel = '5.14.0-570.112.1'
matching = [t for t in tags if kernel in t]
for t in sorted(matching):
print(t)
"Example matching tags for driver 580.159.03 and kernel 5.14.0-570.112.1.el9_6.x86_64:
580.159.03-5.14.0-570.112.1.el9_6.x86_64-rhcos4.19
580.159.03-5.14.0-570.112.1.el9_6.x86_64-rhcos4.20
580.159.03-5.14.0-570.112.1.el9_6.x86_64-rhel9_6
Access to registry.stage.redhat.io requires authentication. To pull or inspect images with skopeo, pass --creds '<user>:<password>'.
To use this registry directly from the cluster (e.g., if the OS tag issue is fixed), create an image pull secret in the GPU operator namespace:
oc create secret docker-registry gpu-driver-registry \
--docker-server=registry.stage.redhat.io \
--docker-username='<user>' \
--docker-password='<password>' \
-n nvidia-gpu-operatorThen reference it in the ClusterPolicy driver section:
driver:
imagePullSecrets:
- gpu-driver-registryWhen using usePrecompiled: true, the GPU Operator auto-constructs the image tag as:
{version}-{kernel_version}.{arch}-{os_tag}
The os_tag is derived from the node's /etc/os-release (ID + VERSION_ID). On RHCOS nodes:
ID="rhel"
VERSION_ID="9.6"
OPENSHIFT_VERSION="4.21"
This causes the operator to generate rhel9.6 (with a dot) as the OS tag suffix, producing a tag like:
580.159.03-5.14.0-570.112.1.el9_6.x86_64-rhel9.6
However, the available image tags in registry.stage.redhat.io use either:
rhcos4.20(OpenShift version-based)rhel9_6(with underscore, not dot)
Neither matches the operator-generated tag. This is a mismatch between the GPU Operator's tag construction logic and the Red Hat image naming convention.
Re-tag the image to match the operator-generated tag and push to a public registry:
skopeo copy \
--src-creds '<user>:<password>' \
docker://registry.stage.redhat.io/nvidia/gpu-driver-rhel9:580.159.03-5.14.0-570.112.1.el9_6.x86_64-rhcos4.20 \
docker://quay.io/<your-org>/gpu-driver-rhel9:580.159.03-5.14.0-570.112.1.el9_6.x86_64-rhel9.6Make sure the destination repository is public, or create a pull secret for it.
oc apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
name: nvidia-gpu-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: gpu-operator-og
namespace: nvidia-gpu-operator
spec:
targetNamespaces:
- nvidia-gpu-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: gpu-operator-certified
namespace: nvidia-gpu-operator
spec:
channel: v26.3
name: gpu-operator-certified
source: certified-operators
sourceNamespace: openshift-marketplace
installPlanApproval: Automatic
EOFWait for the CSV to succeed:
oc get csv -n nvidia-gpu-operator -woc apply -f - <<'EOF'
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
name: gpu-cluster-policy
spec:
operator:
defaultRuntime: crio
use_ocp_driver_toolkit: true
daemonsets: {}
driver:
enabled: true
usePrecompiled: true
repository: quay.io/<your-org>
image: gpu-driver-rhel9
version: "580.159.03"
upgradePolicy:
autoUpgrade: true
maxParallelUpgrades: 1
maxUnavailable: "25%"
devicePlugin:
enabled: true
dcgm:
enabled: true
dcgmExporter:
enabled: true
gfd:
enabled: true
migManager:
enabled: false
nodeStatusExporter:
enabled: true
toolkit:
enabled: true
validator:
plugin:
env:
- name: WITH_WORKLOAD
value: "false"
EOFWait for the ClusterPolicy to become ready:
watch oc get clusterpolicy gpu-cluster-policy -o jsonpath='{.status.state}'oc debug node/<gpu-node> -- chroot /host mokutil --sb-state
# Expected: SecureBoot enabledoc debug node/<gpu-node> -- chroot /host modinfo -F signer nvidia
# Expected: Red Hat Enterprise Linux kernel signing keyoc get node -l feature.node.kubernetes.io/pci-10de.present=true \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}nvidia.com/gpu: {.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'oc apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: gpu-burn-test
namespace: default
spec:
restartPolicy: Never
containers:
- name: gpu-burn
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
resources:
limits:
nvidia.com/gpu: 1
EOF
oc wait --for=jsonpath='{.status.phase}'=Succeeded pod/gpu-burn-test --timeout=120s
oc logs gpu-burn-test
# Expected: Test PASSEDoc delete pod gpu-burn-test -n default --ignore-not-found| Component | Version |
|---|---|
| OpenShift | 4.21.14 |
| Kubernetes | 1.34.6 |
| RHCOS | 9.6.20260504-0 |
| Kernel | 5.14.0-570.112.1.el9_6.x86_64 |
| CRI-O | 1.34.7-2.rhaos4.21.git7d1fe5c.el9 |
| NFD Operator | 4.21.0-202605200241 |
| GPU Operator | 26.3.1 (channel v26.3) |
| NVIDIA Driver | 580.159.03 (precompiled, signed) |
| CUDA | 13.2.1 |
| AWS Instance Type | g4dn.xlarge (NVIDIA T4) |
| AWS Region | us-west-2 |