Installing OpenShift SR-IOV Operator on SNO (OpenShift v4.16.x)

Part I: Prerequisites for SR-IOV Operator (OpenShift 4.16.x)

Gather details about attached network interfaces. You can run this command "as-is" to return extremely useful information that we will reference several times throughout this and other guides. There's an example of what should be returned below.
- Use the following link to gather network interface information (using oc debug commands). Be sure that your KUBECONFIG variable has been set first, and that you have debug priviledges to the cluster (common for the kubeadmin user). These instructions and example output can be found HERE.
IMPORTANT: You must validate the hardare (NICs) you're using with the SR-IOV compatibility list. There are options/methods to use SR-IOV operator with unsupported NICs, but this needs to be worked on direclty with Red Hat.

Required: Deploy a Performance Profile

Follow the instructions for deploying a PerformanceProfile.

Optional: Install NMState Operator (OpenShift 4.16)

It is at this point where I usually recommend that you deploy the NMState Operator (it's optional, but highly recommended).

Part II: Install SI-IOV Operator (OpenShift 4.16.x)

Apply the following Namespace, OperatorGroup, and Subscription. This will install the SR-IOV Operator. In later steps, you will deploy a couple of CRs which will configure the operator and thus, configure your SR-IOV-based NICs.

oc apply -f - <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    openshift.io/cluster-monitoring: "true"
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
    security.openshift.io/scc.podSecurityLabelSync: "true"
  name: openshift-sriov-network-operator

---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sriov-network-operators
  namespace: openshift-sriov-network-operator
spec:
  targetNamespaces:
  - openshift-sriov-network-operator
  upgradeStrategy: Default

---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/sriov-network-operator.openshift-sriov-network-operator: ""
  name: sriov-network-operator
  namespace: openshift-sriov-network-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: sriov-network-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

Now we need to install an SriovOperatorConfig, which is going to tell the operator important things like logging levels, how you want to treat reboots, etc. This should work for PoCs, but you want to pay attention to one option (disableDrain) to make sure you're ok with the node rebooting. If you need to really lock down reboots (i.e. similar to production envirnments), then change the setting accordingly.
```
oc apply -f - <<EOF
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovOperatorConfig
metadata:
  name: default
  namespace: openshift-sriov-network-operator
spec:
  enableInjector: true
  enableOperatorWebhook: true
  disableDrain: true
  logLevel: 2
EOF
```

SR-IOV Deployment Overview

With the two operators out of the way, and configured at a global level, it's time to take the infroamtion that you completed in Step 1 and apply them here; to the deployment. Deployments for SR-IOV-based resources are namespaced and that means that you can give a specific number of VFs to a given namespace. Namespaces can have their own resources. This is important when creating your deployments. I will use an example deployment, but you will need to edit some information to personalize these deployments. I will help you identify the changes needed as we go along.

Label the node(s) with the following label. This is required.
```
NODE_NAME=$(oc get no -o name)

oc label $NODE_NAME feature.node.kubernetes.io/network-sriov.capable=true
```
NOTE: *The command above assumes that this is for a SNO deployment, however if you need this to be applied for a non-SNO deployment, simply replace the $NODE_NAME with the name of the node you want to enable SR-IOV on, like node/<node-name>.

With the nodes labeled, you will need to create the following manifest/deployment using the NIC details from our command earlier.

INTERFACE=ens8f0
VF_RANGE=0-31
VF_TOTAL=64
VLAN=3
VENDOR=8086
DEVICE_ID=158b
DEVICE_TYPE=vfio-pci
RDMA=false
PRIORITY=99
NAME=policy-sriov-$INTERFACE-0031
NAMESPACE=jinkit-vms
RESOURCE=${INTERFACE}_0031

oc apply -f - <<EOF | tee /dev/tty
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: $NAME
  namespace: openshift-sriov-network-operator
spec:
  deviceType: $DEVICE_TYPE
  isRdma: $RDMA
  nicSelector:
    deviceID: "$DEVICE_ID"
    vendor: "$VENDOR"
    pfNames:
    - $INTERFACE#$VF_RANGE
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  numVfs: $VF_TOTAL
  priority: $PRIORITY
  resourceName: $RESOURCE

---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: $INTERFACE-vlan3-d
  namespace: openshift-sriov-network-operator
spec:
  ipam: |-
    {
      "ipam": {
        "type": "dhcp"
      }
    }
  networkNamespace: $NAMESPACE
  resourceName: $RESOURCE
  vlan: $VLAN
EOF

Example SR-IOV Deployment

oc apply -f - <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: panos-vms
  name: panos-vms
spec: {}

---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: policy-sriov-mcx4-enp4s0f1np1
  namespace: openshift-sriov-network-operator
spec:
  deviceType: vfio-pci
  isRdma: false
  nicSelector:
    vendor: 15b3
    deviceID: '1015'
    pfNames:
      - enp4s0f1np1#0-5
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: 'true'
  numVfs: 6
  priority: 97
  resourceName: mcx4_enp4s0f1np1_003

---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: panos-vms-vlan3-d
  namespace: openshift-sriov-network-operator
spec:
  ipam: |-
    {
      "ipam": {
        "type": "dhcp"
      }
    }
  logLevel: info
  networkNamespace: panos-vms
  resourceName: mcx4_enp4s0f1np1_003
  vlan: 3

---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: panos-vms-vlan3-w
  namespace: openshift-sriov-network-operator
spec:
  ipam: |-
    {
      "type": "whereabouts",
      "range": "192.168.4.0/24",
      "range_start": "192.168.4.160",
      "range_end": "192.168.4.170",
      "routes": [{"dst":"192.168.0.0/16","gw":"192.168.4.1"}],
      "dns": {"nameservers": ["8.8.8.8"]}
    }
  logLevel: info
  networkNamespace: panos-vms
  resourceName: mcx4_enp4s0f1np1_003
  vlan: 3
EOF