GPU / PCI passthru via VFIO

gpu-passthrough-to-vm.bu

# 1. Add "intel_iommu=on" or "amd_iommu=on" to the kernel command line
# 2. Tell vfio-pci to claim your devices in vfio-pci-devices.conf
# 3. Load the vfio-pci module via load-vfio-pci.conf

---
variant: openshift
version: 4.16.0
metadata:
  name: 100-gpu-passthrough-to-vm
  labels:
    machineconfiguration.openshift.io/role: worker  ###Review/Change this

openshift:
  kernel_arguments:                                 ###Review/Change this
    #- amd_iommu=on
    - intel_iommu=on

storage:
  files:
  - path: /etc/modules-load.d/load-vfio-pci.conf 
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci

  - path: /etc/modprobe.d/vfio-pci-devices.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=10de:20b5             ### Review/Change this
        # NOTE:
        #   You can specify multiple card types as a comma-separated value to ids=
        #   NVIDIA TU104GL [Tesla T4] == 10de:1eb8
        #   NVIDIA GA102GL [A40]      == 10de:2235
        # options vfio-pci ids=10de:1eb8,10de:2235

Convert the butane file above into YAML using this procedure

curl -O https://mirror.openshift.com/pub/openshift-v4/clients/butane/latest/butane

chmod a+x ./butane

./butane --version
Butane 0.24.0

./butane gpu-passthrough-to-vm.bu -o gpu-passthrough-to-vm.yaml

oc create -f gpu-passthrough-to-vm.yaml

Tell OpenShift Virtualization (kubevirt-hyperconverged) that it can expose the GPU

oc patch -n openshift-cnv hco/kubevirt-hyperconverged --type merge -p '{"spec":{"permittedHostDevices":{"pciHostDevices":[{"pciDeviceSelector":"10de:20b5","resourceName":"nvidia.com/A100"}]}}}'

Update your Nodepool so that the Hosted Control Plane VMs are created with a GPU

oc patch -n NAMESPACE \
  nodepool/NODE_POOL_NAME --type merge -p '{"spec": {"platform": {"kubevirt": {"hostDevices": [{"count": 1, "deviceName": "nvidia.com/A100"}]}}}}'

This is what it looks like...

spec:
  platform:
    kubevirt:
      hostDevices:
      - count: 1
        deviceName: nvidia.com/A100