-
I always recommend to install the OpenShift NMState Operator when installing the SR-IOV Operator, and this is primarily so that administrators can manage/configure NICs using a declarative
nmclimodel. Additionally, administrators will notice that the NMState operator will add additional WebUI dashboards under the Network navigation pane (NetworkAttachmentDefinitions, NodeNetworkConfigurationPolicy, and NodeNetworkState). *NodeNetworkState is extremely useful in determining NIC topology and other information.Although this is optional, I would recommend installing the NMSTate operator by deploying the following manifests.
oc apply -f - <<EOF --- apiVersion: v1 kind: Namespace metadata: labels: kubernetes.io/metadata.name: openshift-nmstate name: openshift-nmstate pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/warn: privileged security.openshift.io/scc.podSecurityLabelSync: "true" name: openshift-nmstate --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: annotations: olm.providedAPIs: NMState.v1.nmstate.io name: openshift-nmstate namespace: openshift-nmstate spec: targetNamespaces: - openshift-nmstate upgradeStrategy: Default --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: labels: operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate: "" name: kubernetes-nmstate-operator namespace: openshift-nmstate spec: channel: stable installPlanApproval: Automatic name: kubernetes-nmstate-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
-
Now you want to configure a
defaultNMState CR, with no further configuration. This will configure the operator to deploy an NMState management environment, which we can use later to configure interfaces if we need to.oc apply -f - <<EOF apiVersion: nmstate.io/v1 kind: NMState metadata: name: nmstate spec: {} EOF
-
After the NMState operator has been successfully installed, you will get a pop-up message informing you to refresh the webpage. When this happens, click the blue hyperlink in the pop-up window to perform this refresh. You can manually refresh the page as well, if that's what you prefer.
-
Run the following interface script from this LINK. Follow the instructions carefully. You will have output that looks similar to the following example below.
Starting pod/roderika-debug-jqzb6 ... To use host binaries, run `chroot /host` Pod IP: 192.168.3.99 If you don't see a command prompt, try pressing enter. Interface IP Address MAC Address Link State Link Speed MTU Vendor Device ID Driver Total VFs Configured VFs PCIe Address eno1np0 - b0:7b:25:d5:98:fa UP 10000Mb/s 9000 14e4 16d8 bnxt_en 8 0 0000:18:00.0 eno2np1 - b0:7b:25:d5:98:fa UP 10000Mb/s 9000 14e4 16d8 bnxt_en 8 0 0000:18:00.1 enp0s20f0u14u3 169.254.1.2 b0:7b:25:d5:99:0d UNKNOWN 425Mb/s 1500 - - cdc_ether - - usb-0000:00:14.0-14.3 ens1f0 - 6c:fe:54:5a:32:78 DOWN - 1500 8086 1592 ice 128 0 0000:5e:00.0 ens1f1 - 6c:fe:54:5a:32:79 DOWN - 1500 8086 1592 ice 128 0 0000:5e:00.1 ens8f0 - 40:a6:b7:46:87:40 UP 10000Mb/s 9000 8086 158b i40e 64 0 0000:b1:00.0 ens8f1 - 40:a6:b7:46:87:41 UP 10000Mb/s 9000 8086 158b i40e 64 0 0000:b1:00.1 Removing debug pod ...
In the case of the example below, the interfaces I really want to focus on are
ens1f0.ens1f0 - 6c:fe:54:5a:32:78 DOWN - 1500 8086 1592 ice 128 0 0000:5e:00.0 ens1f1 - 6c:fe:54:5a:32:79 DOWN - 1500 8086 1592 ice 128 0 0000:5e:00.1
This is an Intel E810-CQDA2 (100G x2) NIC. I can tell this because it uses the ICE driver. But if you look at some other details about this NIC, you will notice that there's no link speed and that the MTU is set to 1500. We will make these changes in the following steps.
-
Please have a look at the following example. We are applying the following
NodeNetworkConfigurationPolicyto simply enable and configure MTU settings for the Intel E810-CQDA2 (100G x2) NIC. This manifest looks very similar to anNMStateConfigobject. See the differences down belowNOTE: Your environment will be different! DO NOT BLINDLY COPY AND PASTE!!!
oc apply -f - <<EOF --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: e810-policy-roderika spec: nodeSelector: kubernetes.io/hostname: "roderika" desiredState: interfaces: - name: ens1f0 description: Standard Ethernet via E810-100G NIC port 0 type: ethernet state: up mtu: 9000 ipv4: enabled: false ipv6: enabled: false - name: ens1f1 description: Standard Ethernet via E810-100G NIC port 1 type: ethernet state: up mtu: 9000 ipv4: enabled: false ipv6: enabled: false EOF
You can verify that the NNCP was configured correctly by looking at the YAML output of the object, either on the CLI or via the webUI (for this, go to Home > API Explorer and search for NodeNetworkConfigurationPolicy). In the output, look at the status field. For our example, you want to see the following result: "message": "1/1 nodes successfully configured", which means that the deployment was successful.
Example:
oc get nncp e810-policy-roderika -o jsonpath="{.status.conditions}" | jq .Results:
[
{
"lastHeartbeatTime": "2024-09-06T13:42:04Z",
"lastTransitionTime": "2024-09-06T13:42:04Z",
"message": "1/1 nodes successfully configured",
"reason": "SuccessfullyConfigured",
"status": "True",
"type": "Available"
},
{
"lastHeartbeatTime": "2024-09-06T13:42:04Z",
"lastTransitionTime": "2024-09-06T13:42:04Z",
"reason": "SuccessfullyConfigured",
"status": "False",
"type": "Degraded"
},
{
"lastHeartbeatTime": "2024-09-06T13:42:04Z",
"lastTransitionTime": "2024-09-06T13:42:04Z",
"reason": "ConfigurationProgressing",
"status": "False",
"type": "Progressing"
}
]You can also use the following jq example to verify that the MTU has changed on the interfaces ens1f0 and ens1f.
Example:
oc get NodeNetworkState roderika -o json | jq '.status.currentState.interfaces[] | select(.name == "ens1f0" or .name == "ens1f1") | {name, mtu}'Results:
{
"name": "ens1f0",
"mtu": 9000
}
{
"name": "ens1f1",
"mtu": 9000
}One key very important and major between an NMStateConfig, which you are likely already familiar with, and a NodeNetworkConfigurationPolicy(NNCP) is that the NodeNetworkConfigurationPolicy is asymetric in nature. Applying changes like described above will work perfectly fine, however if you need to remove the changes (due to a misconfiguration, etc), you will need to intentionally reverse the changes in your NNCP and reapply them. Deleteing NNCP deployments will NOT revert the changes back to their original settings.
-
For example, if I wanted to adjust the MTU from our applied changes above (with a value of 9000) to an MTU of 1500 (the original values), simply deleting the NNCP will not work. I would need to apply the following manifest to reverse the changes.
oc apply -f - <<EOF --- apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: e810-policy-roderika spec: nodeSelector: kubernetes.io/hostname: "roderika" desiredState: interfaces: - name: ens1f0 type: ethernet state: down mtu: 1500 ipv4: enabled: false ipv6: enabled: false - name: ens1f1 type: ethernet state: down mtu: 1500 ipv4: enabled: false ipv6: enabled: false EOF
-
Then you will need to wait until the changes are applied successfully (like our previous steps above). You can loop the previous command if you'd prefer.
Looping the status field for the NNCP:
while sleep 1; do oc get nncp e810-policy-roderika -o jsonpath="{.status.conditions}" | jq . ; done
This means that it is still progressing:
[ { "lastHeartbeatTime": "2024-09-06T14:46:01Z", "lastTransitionTime": "2024-09-06T14:46:01Z", "status": "Unknown", "type": "Available" }, { "lastHeartbeatTime": "2024-09-06T14:46:01Z", "lastTransitionTime": "2024-09-06T14:46:01Z", "status": "Unknown", "type": "Degraded" }, { "lastHeartbeatTime": "2024-09-06T14:46:01Z", "lastTransitionTime": "2024-09-06T14:46:01Z", "status": "Unknown", "type": "Progressing" } ]This means that it was applied successfully:
[ { "lastHeartbeatTime": "2024-09-06T14:52:57Z", "lastTransitionTime": "2024-09-06T14:52:57Z", "message": "1/1 nodes successfully configured", "reason": "SuccessfullyConfigured", "status": "True", "type": "Available" }, { "lastHeartbeatTime": "2024-09-06T14:52:57Z", "lastTransitionTime": "2024-09-06T14:52:57Z", "reason": "SuccessfullyConfigured", "status": "False", "type": "Degraded" }, { "lastHeartbeatTime": "2024-09-06T14:52:57Z", "lastTransitionTime": "2024-09-06T14:52:57Z", "reason": "ConfigurationProgressing", "status": "False", "type": "Progressing" } ] -
Now you can successfully remove the NNCP object entirely.
❯ oc get nncp NAME STATUS REASON e810-policy-roderika Available SuccessfullyConfigured ❯ oc delete nncp e810-policy-roderika nodenetworkconfigurationpolicy.nmstate.io "e810-policy-roderika" deleted