Created
April 2, 2025 18:02
-
-
Save surajssd/7962610ac5c0dd77cabc9c5bab949641 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ kubectl -n network-operator exec -it mofed-ubuntu22.04-54cb554cbd-ds-cmwmf -- cat /tmp/entrypoint_debug_cmds.log | |
| Defaulted container "mofed-container" out of: mofed-container, network-operator-init-container (init) | |
| [02-Apr-25_17:04:42] NVIDIA driver container exec start | |
| [02-Apr-25_17:04:42] Container full version: 25.01-0.6.0.0-0 | |
| [02-Apr-25_17:04:42] Verifying loaded modules will not prevent future driver restart | |
| [02-Apr-25_17:04:42] Executing driver sources container | |
| [02-Apr-25_17:04:42] Drivers inventory path is set: /mnt/drivers-inventory | |
| [02-Apr-25_17:04:42] Unsetting driver ready state | |
| [02-Apr-25_17:04:42] Query VFs info from [1] devices | |
| [02-Apr-25_17:04:42] Query representors info from [1] devices | |
| [02-Apr-25_17:04:53] Starting driver build | |
| [02-Apr-25_17:08:20] Stored driver packages at /mnt/drivers-inventory/5.15.0-1082-azure/25.01-0.6.0.0 | |
| [02-Apr-25_17:08:20] Storing the check sum for build artifacts at /mnt/drivers-inventory/5.15.0-1082-azure/25.01-0.6.0.0.checksum, check sum: 5cddee8c2fa24e4ecd45a5e0469c567f | |
| [02-Apr-25_17:08:21] Apply blacklisted mofed modules file to host (/etc/modprobe.d/blacklist-ofed-modules.conf) | |
| [02-Apr-25_17:08:39] Remove blacklisted mofed modules file from host | |
| [02-Apr-25_17:08:39] Current mlx5_core driver version: 25.01-0.6.0 | |
| [02-Apr-25_17:08:39] Mounting Mellanox OFED driver container shared kernel headers | |
| [02-Apr-25_17:08:39] Setting driver ready state | |
| [02-Apr-25_17:08:39] NVIDIA driver container exec end, sleeping |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ kubectl logs -n network-operator mofed-ubuntu22.04-54cb554cbd-ds-cmwmf > mofed-ubuntu22.04-54cb554cbd-ds-cmwmf.logs | |
| Defaulted container "mofed-container" out of: mofed-container, network-operator-init-container (init) | |
| [02-Apr-25_17:04:42] NVIDIA driver container exec start | |
| [02-Apr-25_17:04:42] Container full version: 25.01-0.6.0.0-0 | |
| [02-Apr-25_17:04:42] Verifying loaded modules will not prevent future driver restart | |
| [02-Apr-25_17:04:42] Executing driver sources container | |
| [02-Apr-25_17:04:42] Drivers inventory path is set: /mnt/drivers-inventory | |
| [02-Apr-25_17:04:42] Unsetting driver ready state | |
| [02-Apr-25_17:04:42] Query VFs info from [1] devices | |
| [02-Apr-25_17:04:42] Query representors info from [1] devices | |
| Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] | |
| Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB] | |
| Get:3 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [47.7 kB] | |
| Get:4 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [3972 kB] | |
| Get:5 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1241 kB] | |
| Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2773 kB] | |
| Get:7 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] | |
| Get:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] | |
| Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB] | |
| Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB] | |
| Get:11 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB] | |
| Get:12 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB] | |
| Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1540 kB] | |
| Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [55.7 kB] | |
| Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3081 kB] | |
| Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [4126 kB] | |
| Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [82.7 kB] | |
| Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [35.2 kB] | |
| Fetched 37.3 MB in 3s (12.2 MB/s) | |
| Reading package lists... | |
| debconf: unable to initialize frontend: Dialog | |
| debconf: (TERM is not set, so the dialog frontend is not usable.) | |
| debconf: falling back to frontend: Readline | |
| debconf: unable to initialize frontend: Readline | |
| debconf: (This frontend requires a controlling tty.) | |
| debconf: falling back to frontend: Teletype | |
| dpkg-preconfigure: unable to re-open stdin: | |
| Reading package lists... | |
| Building dependency tree... | |
| Reading state information... | |
| pkg-config is already the newest version (0.29.2-1ubuntu3). | |
| The following NEW packages will be installed: | |
| linux-azure-headers-5.15.0-1082 linux-headers-5.15.0-1082-azure | |
| 0 upgraded, 2 newly installed, 0 to remove and 19 not upgraded. | |
| Need to get 15.0 MB of archives. | |
| After this operation, 102 MB of additional disk space will be used. | |
| Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-azure-headers-5.15.0-1082 all 5.15.0-1082.91 [12.3 MB] | |
| Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-1082-azure amd64 5.15.0-1082.91 [2632 kB] | |
| Fetched 15.0 MB in 2s (9302 kB/s) | |
| Selecting previously unselected package linux-azure-headers-5.15.0-1082. | |
| (Reading database ... | |
| (Reading database ... 5% | |
| (Reading database ... 10% | |
| (Reading database ... 15% | |
| (Reading database ... 20% | |
| (Reading database ... 25% | |
| (Reading database ... 30% | |
| (Reading database ... 35% | |
| (Reading database ... 40% | |
| (Reading database ... 45% | |
| (Reading database ... 50% | |
| (Reading database ... 55% | |
| (Reading database ... 60% | |
| (Reading database ... 65% | |
| (Reading database ... 70% | |
| (Reading database ... 75% | |
| (Reading database ... 80% | |
| (Reading database ... 85% | |
| (Reading database ... 90% | |
| (Reading database ... 95% | |
| (Reading database ... 100% | |
| (Reading database ... 46485 files and directories currently installed.) | |
| Preparing to unpack .../linux-azure-headers-5.15.0-1082_5.15.0-1082.91_all.deb ... | |
| Unpacking linux-azure-headers-5.15.0-1082 (5.15.0-1082.91) ... | |
| Selecting previously unselected package linux-headers-5.15.0-1082-azure. | |
| Preparing to unpack .../linux-headers-5.15.0-1082-azure_5.15.0-1082.91_amd64.deb ... | |
| Unpacking linux-headers-5.15.0-1082-azure (5.15.0-1082.91) ... | |
| Setting up linux-azure-headers-5.15.0-1082 (5.15.0-1082.91) ... | |
| Setting up linux-headers-5.15.0-1082-azure (5.15.0-1082.91) ... | |
| /etc/kernel/header_postinst.d/dkms: | |
| * dkms: running auto installation service for kernel 5.15.0-1082-azure | |
| ...done. | |
| [02-Apr-25_17:04:53] Starting driver build | |
| Logs dir: /tmp/OFED.363.logs | |
| General log file: /tmp/OFED.363.logs/general.log | |
| [33mUnsupported package: kmp[0m | |
| [32m | |
| Below is the list of OFED packages that you have chosen | |
| (some may have been added by the installer due to package dependencies): | |
| [0m | |
| mlnx-tools | |
| mlnx-ofed-kernel-utils | |
| mlnx-ofed-kernel-modules | |
| iser-modules | |
| isert-modules | |
| srp-modules | |
| mlnx-nfsrdma-modules | |
| mlnx-nvme-modules | |
| kernel-mft-modules | |
| knem-modules | |
| xpmem | |
| xpmem-modules | |
| This program will install the OFED package on your machine. | |
| Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed. | |
| Those packages are removed due to conflicts with OFED, do not reinstall them. | |
| Installing new packages | |
| Building DEB for mlnx-tools-25.01 (mlnx-tools)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc | |
| Building DEB for mlnx-ofed-kernel-utils-25.01.OFED.25.01.0.6.0.1 (mlnx-ofed-kernel)... | |
| -W- --with-mlx5-ipsec is enabled | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Extracting mlnx-ofed-kernel sources to /var/tmp/ofed_src ... | |
| dpkg -x '/run/mellanox/src/MLNX_OFED_SRC-25.01-0.6.0.0/DEBS/ubuntu22.04/x86_64/mlnx-ofed-kernel-modules_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb' /var/tmp/ofed_src >/dev/null | |
| Building DEB for iser-modules-25.01.OFED.25.01.0.6.0.1 (iser)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Building DEB for isert-modules-25.01.OFED.25.01.0.6.0.1 (isert)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Building DEB for srp-modules-25.01.OFED.25.01.0.6.0.1 (srp)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Building DEB for mlnx-nfsrdma-modules-25.01.OFED.25.01.0.6.0.1 (mlnx-nfsrdma)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Building DEB for mlnx-nvme-modules-25.01.OFED.25.01.0.6.0.1 (mlnx-nvme)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Building DEB for kernel-mft-modules-4.31.0.149 (kernel-mft)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Building DEB for knem-modules-1.1.4.90mlnx3 (knem)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| Building DEB for xpmem-2.7.4 (xpmem)... | |
| Running /usr/bin/dpkg-buildpackage -us -uc -Pmodules | |
| [32mBuild passed successfully[0m | |
| [02-Apr-25_17:08:20] Stored driver packages at /mnt/drivers-inventory/5.15.0-1082-azure/25.01-0.6.0.0 | |
| [02-Apr-25_17:08:20] Storing the check sum for build artifacts at /mnt/drivers-inventory/5.15.0-1082-azure/25.01-0.6.0.0.checksum, check sum: 5cddee8c2fa24e4ecd45a5e0469c567f | |
| Selecting previously unselected package iser-modules. | |
| (Reading database ... 70712 files and directories currently installed.) | |
| Preparing to unpack .../iser-modules_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking iser-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package isert-modules. | |
| Preparing to unpack .../isert-modules_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking isert-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package kernel-mft-modules. | |
| Preparing to unpack .../kernel-mft-modules_4.31.0.149-1.kver.5.15.0-1082-azure_all.deb ... | |
| Unpacking kernel-mft-modules (4.31.0.149-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package knem-modules. | |
| Preparing to unpack .../knem-modules_1.1.4.90mlnx3-OFED.23.10.0.2.1.1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking knem-modules (1.1.4.90mlnx3-OFED.23.10.0.2.1.1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package knem. | |
| Preparing to unpack .../knem_1.1.4.90mlnx3-OFED.23.10.0.2.1.1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking knem (1.1.4.90mlnx3-OFED.23.10.0.2.1.1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package mlnx-nfsrdma-modules. | |
| Preparing to unpack .../mlnx-nfsrdma-modules_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking mlnx-nfsrdma-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package mlnx-nvme-modules. | |
| Preparing to unpack .../mlnx-nvme-modules_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking mlnx-nvme-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package mlnx-ofed-kernel-modules. | |
| Preparing to unpack .../mlnx-ofed-kernel-modules_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking mlnx-ofed-kernel-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package mlnx-ofed-kernel-utils. | |
| Preparing to unpack .../mlnx-ofed-kernel-utils_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking mlnx-ofed-kernel-utils (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package mlnx-tools. | |
| Preparing to unpack .../mlnx-tools_25.01-0.2501060_amd64.deb ... | |
| Unpacking mlnx-tools (25.01-0.2501060) ... | |
| Selecting previously unselected package srp-modules. | |
| Preparing to unpack .../srp-modules_25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking srp-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package xpmem-modules:amd64. | |
| Preparing to unpack .../xpmem-modules_2.7.4-1.2501060.kver.5.15.0-1082-azure_amd64.deb ... | |
| Unpacking xpmem-modules:amd64 (2.7.4-1.2501060.kver.5.15.0-1082-azure) ... | |
| Selecting previously unselected package xpmem. | |
| Preparing to unpack .../xpmem_2.7.4-1.2501060.kver.5.15.0-1082-azure_all.deb ... | |
| Unpacking xpmem (2.7.4-1.2501060.kver.5.15.0-1082-azure) ... | |
| Setting up kernel-mft-modules (4.31.0.149-1.kver.5.15.0-1082-azure) ... | |
| Setting up knem-modules (1.1.4.90mlnx3-OFED.23.10.0.2.1.1.kver.5.15.0-1082-azure) ... | |
| Setting up knem (1.1.4.90mlnx3-OFED.23.10.0.2.1.1.kver.5.15.0-1082-azure) ... | |
| Setting up mlnx-tools (25.01-0.2501060) ... | |
| Setting up xpmem-modules:amd64 (2.7.4-1.2501060.kver.5.15.0-1082-azure) ... | |
| Setting up xpmem (2.7.4-1.2501060.kver.5.15.0-1082-azure) ... | |
| Setting up mlnx-ofed-kernel-utils (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Configured /etc/security/limits.conf | |
| Setting up mlnx-ofed-kernel-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| update-alternatives: using /usr/src/ofa_kernel/x86_64/5.15.0-1082-azure to provide /usr/src/ofa_kernel/default (ofa_kernel_headers) in auto mode | |
| Setting up srp-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Setting up iser-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Setting up isert-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Setting up mlnx-nfsrdma-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Setting up mlnx-nvme-modules (25.01.OFED.25.01.0.6.0.1-1.kver.5.15.0-1082-azure) ... | |
| Processing triggers for man-db (2.10.2-1) ... | |
| [02-Apr-25_17:08:21] Apply blacklisted mofed modules file to host (/etc/modprobe.d/blacklist-ofed-modules.conf) | |
| Function: generate_ofed_modules_blacklist | |
| Unloading HCA driver:[60G[ [1;32mOK[0;39m ] | |
| Loading HCA driver and Access Layer:[60G[ [1;32mOK[0;39m ] | |
| [02-Apr-25_17:08:39] Remove blacklisted mofed modules file from host | |
| [02-Apr-25_17:08:39] Current mlx5_core driver version: 25.01-0.6.0 | |
| [02-Apr-25_17:08:39] Mounting Mellanox OFED driver container shared kernel headers | |
| [02-Apr-25_17:08:39] Setting driver ready state | |
| [02-Apr-25_17:08:39] NVIDIA driver container exec end, sleeping |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: apps/v1 | |
| kind: DaemonSet | |
| metadata: | |
| annotations: | |
| deprecated.daemonset.template.generation: "1" | |
| nvidia.network-operator.revision: "688623439" | |
| creationTimestamp: "2025-04-02T17:04:29Z" | |
| generation: 1 | |
| labels: | |
| app: mofed-ubuntu22.04-54cb554cbd | |
| mofed-ds-format-version: "1" | |
| nvidia.com/ofed-driver: "" | |
| nvidia.network-operator.state: state-OFED | |
| name: mofed-ubuntu22.04-54cb554cbd-ds | |
| namespace: network-operator | |
| ownerReferences: | |
| - apiVersion: mellanox.com/v1alpha1 | |
| blockOwnerDeletion: true | |
| controller: true | |
| kind: NicClusterPolicy | |
| name: nic-cluster-policy | |
| uid: 1b4d6b50-bc36-4ee8-917d-a1840b5fe377 | |
| resourceVersion: "4772" | |
| uid: b0c134f7-7664-4376-bdf9-08780eda6937 | |
| spec: | |
| revisionHistoryLimit: 10 | |
| selector: | |
| matchLabels: | |
| app: mofed-ubuntu22.04-54cb554cbd | |
| template: | |
| metadata: | |
| creationTimestamp: null | |
| labels: | |
| app: mofed-ubuntu22.04-54cb554cbd | |
| kernel: 5.15.0-1082-azure | |
| nvidia.com/ofed-driver: "" | |
| spec: | |
| affinity: | |
| nodeAffinity: | |
| requiredDuringSchedulingIgnoredDuringExecution: | |
| nodeSelectorTerms: | |
| - matchExpressions: | |
| - key: feature.node.kubernetes.io/pci-15b3.present | |
| operator: In | |
| values: | |
| - "true" | |
| containers: | |
| - env: | |
| - name: CREATE_IFNAMES_UDEV | |
| value: "true" | |
| - name: NVIDIA_NIC_DRIVERS_INVENTORY_PATH | |
| value: /mnt/drivers-inventory | |
| image: nvcr.io/nvidia/mellanox/doca-driver:25.01-0.6.0.0-0-ubuntu22.04-amd64 | |
| imagePullPolicy: IfNotPresent | |
| livenessProbe: | |
| exec: | |
| command: | |
| - sh | |
| - -c | |
| - lsmod | grep mlx5_core | |
| failureThreshold: 1 | |
| initialDelaySeconds: 30 | |
| periodSeconds: 30 | |
| successThreshold: 1 | |
| timeoutSeconds: 10 | |
| name: mofed-container | |
| readinessProbe: | |
| exec: | |
| command: | |
| - sh | |
| - -c | |
| - lsmod | grep mlx5_core | |
| failureThreshold: 1 | |
| initialDelaySeconds: 10 | |
| periodSeconds: 30 | |
| successThreshold: 1 | |
| timeoutSeconds: 10 | |
| resources: {} | |
| securityContext: | |
| privileged: true | |
| seLinuxOptions: | |
| level: s0 | |
| startupProbe: | |
| exec: | |
| command: | |
| - sh | |
| - -c | |
| - ls /run/mellanox/drivers/.driver-ready | |
| failureThreshold: 60 | |
| initialDelaySeconds: 10 | |
| periodSeconds: 20 | |
| successThreshold: 1 | |
| timeoutSeconds: 10 | |
| terminationMessagePath: /dev/termination-log | |
| terminationMessagePolicy: File | |
| volumeMounts: | |
| - mountPath: /run/mellanox/drivers | |
| mountPropagation: Bidirectional | |
| name: run-mlnx-ofed | |
| - mountPath: /etc/network | |
| name: etc-network | |
| - mountPath: /host/etc | |
| name: host-etc | |
| - mountPath: /host/usr | |
| name: host-usr | |
| - mountPath: /host/lib/udev | |
| name: host-udev | |
| - mountPath: /run/udev | |
| name: host-run-udev | |
| readOnly: true | |
| - mountPath: /host/lib/modules | |
| name: host-lib-modules | |
| - mountPath: /mnt/drivers-inventory | |
| name: drivers-inventory | |
| - mountPath: /etc/modprobe.d/ib_core.conf | |
| name: host-ib-core | |
| dnsPolicy: ClusterFirst | |
| hostNetwork: true | |
| initContainers: | |
| - args: | |
| - --node-name | |
| - $(NODE_NAME) | |
| - --configmap-name | |
| - ofed-init-container-config | |
| - --configmap-namespace | |
| - network-operator | |
| - --configmap-key | |
| - config.json | |
| env: | |
| - name: NODE_NAME | |
| valueFrom: | |
| fieldRef: | |
| apiVersion: v1 | |
| fieldPath: spec.nodeName | |
| image: ghcr.io/mellanox/network-operator-init-container:v0.0.3 | |
| imagePullPolicy: IfNotPresent | |
| name: network-operator-init-container | |
| resources: {} | |
| terminationMessagePath: /dev/termination-log | |
| terminationMessagePolicy: File | |
| nodeSelector: | |
| feature.node.kubernetes.io/kernel-version.full: 5.15.0-1082-azure | |
| feature.node.kubernetes.io/pci-15b3.present: "true" | |
| feature.node.kubernetes.io/system-os_release.ID: ubuntu | |
| feature.node.kubernetes.io/system-os_release.VERSION_ID: "22.04" | |
| priorityClassName: system-node-critical | |
| restartPolicy: Always | |
| schedulerName: default-scheduler | |
| securityContext: {} | |
| serviceAccount: ofed-driver | |
| serviceAccountName: ofed-driver | |
| terminationGracePeriodSeconds: 300 | |
| tolerations: | |
| - effect: NoSchedule | |
| key: nvidia.com/gpu | |
| operator: Exists | |
| volumes: | |
| - hostPath: | |
| path: /run/mellanox/drivers | |
| type: "" | |
| name: run-mlnx-ofed | |
| - hostPath: | |
| path: /etc/network | |
| type: "" | |
| name: etc-network | |
| - hostPath: | |
| path: /etc | |
| type: "" | |
| name: host-etc | |
| - hostPath: | |
| path: /usr | |
| type: "" | |
| name: host-usr | |
| - hostPath: | |
| path: /lib/udev | |
| type: "" | |
| name: host-udev | |
| - hostPath: | |
| path: /run/udev | |
| type: "" | |
| name: host-run-udev | |
| - hostPath: | |
| path: /lib/modules | |
| type: "" | |
| name: host-lib-modules | |
| - hostPath: | |
| path: /var/opt/mofed-container/inventory | |
| type: DirectoryOrCreate | |
| name: drivers-inventory | |
| - hostPath: | |
| path: /etc/modprobe.d/ib_core.conf | |
| type: FileOrCreate | |
| name: host-ib-core | |
| updateStrategy: | |
| type: OnDelete | |
| status: | |
| currentNumberScheduled: 2 | |
| desiredNumberScheduled: 2 | |
| numberAvailable: 2 | |
| numberMisscheduled: 0 | |
| numberReady: 2 | |
| observedGeneration: 1 | |
| updatedNumberScheduled: 2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ helm ls -n network-operator | |
| NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION | |
| network-operator network-operator 1 2025-04-02 10:03:46.343682 -0700 PDT deployed network-operator-25.1.0 v25.1.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: mellanox.com/v1alpha1 | |
| kind: NicClusterPolicy | |
| metadata: | |
| name: nic-cluster-policy | |
| spec: | |
| nodeAffinity: | |
| requiredDuringSchedulingIgnoredDuringExecution: | |
| nodeSelectorTerms: | |
| - matchExpressions: | |
| - key: feature.node.kubernetes.io/pci-15b3.present | |
| operator: In | |
| values: | |
| - "true" | |
| ofedDriver: | |
| forcePrecompiled: false | |
| image: doca-driver | |
| imagePullSecrets: [] | |
| livenessProbe: | |
| initialDelaySeconds: 30 | |
| periodSeconds: 30 | |
| readinessProbe: | |
| initialDelaySeconds: 10 | |
| periodSeconds: 30 | |
| repository: nvcr.io/nvidia/mellanox | |
| startupProbe: | |
| initialDelaySeconds: 10 | |
| periodSeconds: 20 | |
| terminationGracePeriodSeconds: 300 | |
| upgradePolicy: | |
| autoUpgrade: true | |
| drain: | |
| deleteEmptyDir: true | |
| enable: true | |
| force: true | |
| timeoutSeconds: 300 | |
| maxParallelUpgrades: 1 | |
| safeLoad: false | |
| version: 25.01-0.6.0.0-0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ➜ kubectl -n network-operator exec -it -c mofed-container mofed-ubuntu22.04-54cb554cbd-ds-fh4cc -- bash -x /etc/init.d/openibd restart | |
| + OPENIBD_CONFIG=/etc/infiniband/openib.conf | |
| + CONFIG=/etc/infiniband/openib.conf | |
| + export LANG=C | |
| + LANG=C | |
| + '[' '!' -f /etc/infiniband/openib.conf ']' | |
| + . /etc/infiniband/openib.conf | |
| ++ ONBOOT=yes | |
| ++ ALLOW_STOP=yes | |
| ++ FORCE_MODE=no | |
| +++ hostname -s | |
| ++ NODE_DESC=aks-ibnodepool-52537486-vmss000001 | |
| ++ NODE_DESC_TIME_BEFORE_UPDATE=20 | |
| ++ NODE_DESC_UPDATE_TIMEOUT=120 | |
| ++ POST_START_DELAY=0 | |
| ++ RUN_AFFINITY_TUNER=no | |
| ++ RUN_MLNX_TUNE=no | |
| ++ LOAD_EIPOIB=no | |
| ++ RENICE_IB_MAD=no | |
| ++ RUN_SYSCTL=no | |
| ++ ENABLE_FW_TRACER=yes | |
| ++ UMAD_LOAD=yes | |
| ++ UVERBS_LOAD=yes | |
| ++ RDMA_CM_LOAD=yes | |
| ++ RDMA_UCM_LOAD=yes | |
| ++ MLX5_LOAD=yes | |
| ++ IPOIB_LOAD=yes | |
| ++ SRP_LOAD=no | |
| ++ pwd | |
| + CWD=/root | |
| + cd /etc/infiniband | |
| ++ pwd | |
| + WD=/etc/infiniband | |
| + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/usr/bin:/lib/udev | |
| + '[' -e /etc/profile.d/ofed.sh ']' | |
| + systemd_auto=0 | |
| + bootID= | |
| + '[' X '!=' X ']' | |
| ++ date +%s | |
| ++ tr -d '[:space:]' | |
| + start_time=1743616782 | |
| + base=openibd | |
| + link=openibd | |
| + [[ X == \X\a\u\t\o ]] | |
| + [[ 0 -eq 1 ]] | |
| + [[ Xopenibd != \X\o\p\e\n\i\b\d ]] | |
| + RUNMODE=manual | |
| + ONBOOT=yes | |
| + log_msg 'running in manual mode' | |
| + logger -i 'openibd: running in manual mode' | |
| + echo 1743616782 | |
| + modprobe=/sbin/modprobe | |
| + /sbin/modprobe -c | |
| + grep -q '^allow_unsupported_modules *0' | |
| + ALLOW_STOP=yes | |
| + FORCE_MODE=no | |
| + OPENIBD_PRE_START=/etc/infiniband/pre-start-hook.sh | |
| + OPENIBD_POST_START=/etc/infiniband/post-start-hook.sh | |
| + OPENIBD_PRE_STOP=/etc/infiniband/pre-stop-hook.sh | |
| + OPENIBD_POST_STOP=/etc/infiniband/post-stop-hook.sh | |
| + ACTION=restart | |
| + shift | |
| + ORIG_ACTION=restart | |
| + max_ports_num_in_hca=0 | |
| + FORCE=0 | |
| + WARNED_INBOX_LOAD=0 | |
| + '[' Xyes '!=' Xyes ']' | |
| + grep -i 'SuSE Linux' /etc/issue /etc/os-release | |
| + '[' -z '' ']' | |
| + COLUMNS=80 | |
| + '[' -z '' ']' | |
| + '[' -f /etc/sysconfig/init ']' | |
| + BOOTUP=color | |
| + RES_COL=60 | |
| + MOVE_TO_COL='echo -en \033[60G' | |
| + SETCOLOR_SUCCESS='echo -en \033[1;32m' | |
| + SETCOLOR_FAILURE='echo -en \033[1;31m' | |
| + SETCOLOR_WARNING='echo -en \033[1;33m' | |
| + SETCOLOR_NORMAL='echo -en \033[0;39m' | |
| + LOGLEVEL=1 | |
| + is_serial | |
| + '[' '' = serial ']' | |
| + case `tty` in | |
| ++ tty | |
| + return 1 | |
| + '[' color '!=' verbose ']' | |
| + INITLOG_ARGS=-q | |
| + '[' Xmanual == Xmanual ']' | |
| + get_interface_conf_files | |
| + conf_files=/etc/network/interfaces | |
| + grep -w source /etc/network/interfaces | |
| + grep -qvE '^\s*#' | |
| + '[' X/etc/network/interfaces '!=' X ']' | |
| ++ cat /proc/sys/kernel/random/boot_id | |
| ++ sed -e s/-//g | |
| + curr_bootid=819d9c6febfa48b18287a3521e94eb6c | |
| ++ grep -w '^iface' /etc/network/interfaces | |
| ++ cut '-d ' -f2 | |
| ++ tr -d '"|\'\''' | |
| + echo 819d9c6febfa48b18287a3521e94eb6c | |
| ++ uname -r | |
| ++ cut -c -3 | |
| ++ tr -d . | |
| ++ tr -d '[:space:]' | |
| + KPREFIX=51 | |
| + POST_LOAD_MODULES= | |
| + MODULES_LOADED_STATUS=1 | |
| + RUN_SYSCTL=no | |
| + '[' X == Xyes ']' | |
| + '[' X == Xyes ']' | |
| + IPOIB=0 | |
| + '[' Xyes == Xyes ']' | |
| + IPOIB=1 | |
| + '[' Xno == Xyes ']' | |
| + '[' X == Xyes ']' | |
| + '[' X == Xyes ']' | |
| + '[' Xyes == Xyes ']' | |
| + POST_LOAD_MODULES=' rdma_cm' | |
| + '[' Xyes == Xyes ']' | |
| + POST_LOAD_MODULES=' rdma_cm rdma_ucm' | |
| + GEN1_UNLOAD_MODULES='ib_srp_target scsi_target ib_srp kdapltest_module ib_kdapl ib_sdp eth_ipoib ib_useraccess ib_useraccess_cm ib_cm ib_dapl_srv ib_ip2pr ib_ipoib ib_mlnx_bx ib_tavor mod_thh mod_rhh ib_dm_client ib_sa_client ib_client_query ib_poll ib_mad ib_core ib_services' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp rdma_ucm rdma_cm iw_cm ib_cm ib_local_sa findex' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp rdma_ucm rdma_cm iw_cm ib_cm ib_local_sa findex auxiliary mlxdevm mlx5_vdpa mlx5_vfio_pci' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp rdma_ucm rdma_cm iw_cm ib_cm ib_local_sa findex auxiliary mlxdevm mlx5_vdpa mlx5_vfio_pci mlx5_fwctl fwctl mlx5_dpll' | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp rdma_ucm rdma_cm iw_cm ib_cm ib_local_sa findex auxiliary mlxdevm mlx5_vdpa mlx5_vfio_pci mlx5_fwctl fwctl mlx5_dpll ib_sa ib_uverbs ib_umad ib_mad ib_core ib_addr ib_netlink rdma_rxe mlxfw vfio_mdev' | |
| + STATUS_MODULES='rdma_ucm ib_srp qlgc_vnic ib_sdp rdma_cm ib_local_sa findex ib_ipoib mlx5_core mlx5_ib ib_uverbs ib_umad ib_cm ib_core eth_ipoib mlxfw' | |
| + modinfo scsi_transport_srp | |
| + grep depends: | |
| + grep -q compat | |
| + UNLOAD_MODULES='ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp rdma_ucm rdma_cm iw_cm ib_cm ib_local_sa findex auxiliary mlxdevm mlx5_vdpa mlx5_vfio_pci mlx5_fwctl fwctl mlx5_dpll ib_sa ib_uverbs ib_umad ib_mad ib_core ib_addr ib_netlink rdma_rxe mlxfw vfio_mdev scsi_transport_srp' | |
| + STATUS_MODULES='rdma_ucm ib_srp qlgc_vnic ib_sdp rdma_cm ib_local_sa findex ib_ipoib mlx5_core mlx5_ib ib_uverbs ib_umad ib_cm ib_core eth_ipoib mlxfw scsi_transport_srp' | |
| + modinfo cls_flower | |
| + grep depends: | |
| + grep -q compat | |
| + lsmod | |
| + grep cls_flower | |
| + grep -q compat | |
| + ipoib_ha_pidfile=/var/run/ipoib_ha.pid | |
| + srp_daemon_pidfile=/var/run/srp_daemon.pid | |
| + _truescale=/etc/infiniband/truescale.cmds | |
| + UNLOAD_REC_TIMEOUT=100 | |
| + RC=0 | |
| + trap trap_handler 2 9 15 | |
| + [[ restart =~ force-.* ]] | |
| + '[' Xno == Xyes ']' | |
| + case $ACTION in | |
| + '[' '!' -z /etc/infiniband/pre-stop-hook.sh ']' | |
| + '[' -x /etc/infiniband/pre-stop-hook.sh ']' | |
| + stop | |
| + '[' Xmanual '!=' Xauto ']' | |
| + local cannot_continue=0 | |
| + local blocking_modules= | |
| + grep -q ko2iblnd /proc/modules | |
| + is_active_vf | |
| + lspci | |
| + grep Mellanox | |
| + grep Virtual | |
| + '[' 0 -ne 0 ']' | |
| + virsh -v | |
| + '[' 127 -ne 0 ']' | |
| + return 1 | |
| + for serv in ibacm srp_daemon ibacm.socket | |
| + systemctl is-active --quiet ibacm | |
| + for serv in ibacm srp_daemon ibacm.socket | |
| + systemctl is-active --quiet srp_daemon | |
| + for serv in ibacm srp_daemon ibacm.socket | |
| + systemctl is-active --quiet ibacm.socket | |
| + local 'apps=opensm osmtest ibbs ibns ibacm' | |
| + local pid | |
| + for app in $apps | |
| + /usr/bin/pgrep opensm | |
| + for app in $apps | |
| + /usr/bin/pgrep osmtest | |
| + for app in $apps | |
| + /usr/bin/pgrep ibbs | |
| + for app in $apps | |
| + /usr/bin/pgrep ibns | |
| + for app in $apps | |
| + /usr/bin/pgrep ibacm | |
| + local entries | |
| + '[' -d /dev/infiniband ']' | |
| + '[' -n '' ']' | |
| ++ pidof iscsid | |
| ++ wc -w | |
| + '[' 0 -gt 0 ']' | |
| + is_module ib_srp | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_srp | |
| + RC=1 | |
| + return 1 | |
| + for mod in ib_isert nvme_rdma nvmet_rdma rpcrdma xprtrdma ib_srpt | |
| + is_module ib_isert | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_isert | |
| + RC=1 | |
| + return 1 | |
| + for mod in ib_isert nvme_rdma nvmet_rdma rpcrdma xprtrdma ib_srpt | |
| + is_module nvme_rdma | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w nvme_rdma | |
| + RC=1 | |
| + return 1 | |
| + for mod in ib_isert nvme_rdma nvmet_rdma rpcrdma xprtrdma ib_srpt | |
| + is_module nvmet_rdma | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w nvmet_rdma | |
| + RC=1 | |
| + return 1 | |
| + for mod in ib_isert nvme_rdma nvmet_rdma rpcrdma xprtrdma ib_srpt | |
| + is_module rpcrdma | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rpcrdma | |
| + RC=1 | |
| + return 1 | |
| + for mod in ib_isert nvme_rdma nvmet_rdma rpcrdma xprtrdma ib_srpt | |
| + is_module xprtrdma | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w xprtrdma | |
| + RC=1 | |
| + return 1 | |
| + for mod in ib_isert nvme_rdma nvmet_rdma rpcrdma xprtrdma ib_srpt | |
| + is_module ib_srpt | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_srpt | |
| + RC=1 | |
| + return 1 | |
| + '[' 0 -eq 1 ']' | |
| ++ cat /sys/class/net/bonding_masters | |
| + grep -q '^alias netdev-ib' /etc/modprobe.d/ib_ipoib.conf | |
| + sed -r -i -e 's/(^alias netdev-ib.*)/#\1/' /etc/modprobe.d/ib_ipoib.conf | |
| + is_module mlx4_vnic | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx4_vnic | |
| + RC=1 | |
| + return 1 | |
| + '[' -f /var/run/ipoib_ha.pid ']' | |
| + '[' -n '' ']' | |
| + '[' -f /var/run/ipoib_ha.pid ']' | |
| + '[' -n '' ']' | |
| + '[' -d /sys/class/infiniband_qlgc_vnic/ ']' | |
| + '[' -f /sbin/mlxfc ']' | |
| + '[' 'ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp rdma_ucm rdma_cm iw_cm ib_cm ib_local_sa findex auxiliary mlxdevm mlx5_vdpa mlx5_vfio_pci mlx5_fwctl fwctl mlx5_dpll ib_sa ib_uverbs ib_umad ib_mad ib_core ib_addr ib_netlink rdma_rxe mlxfw vfio_mdev scsi_transport_srp' '!=' '' ']' | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_mthca | |
| + local mod=ib_mthca | |
| + local unload_log | |
| + is_module ib_mthca | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_mthca | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx5_fpga_tools | |
| + local mod=mlx5_fpga_tools | |
| + local unload_log | |
| + is_module mlx5_fpga_tools | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_fpga_tools | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx5_ib | |
| + local mod=mlx5_ib | |
| + local unload_log | |
| + is_module mlx5_ib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_ib | |
| + RC=0 | |
| + return 0 | |
| + case $mod in | |
| + unload_rec mlx5_ib | |
| + local mod=mlx5_ib | |
| + shift | |
| + is_module mlx5_ib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_ib | |
| + RC=0 | |
| + return 0 | |
| + /sbin/modprobe -r mlx5_ib | |
| + '[' 0 -ne 0 ']' | |
| + is_module mlx5_ib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_ib | |
| + RC=1 | |
| + return 1 | |
| + sleep 2 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx5_core | |
| + local mod=mlx5_core | |
| + local unload_log | |
| + is_module mlx5_core | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_core | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx4_ib | |
| + local mod=mlx4_ib | |
| + local unload_log | |
| + is_module mlx4_ib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx4_ib | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_ipath | |
| + local mod=ib_ipath | |
| + local unload_log | |
| + is_module ib_ipath | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_ipath | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ipath_core | |
| + local mod=ipath_core | |
| + local unload_log | |
| + is_module ipath_core | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ipath_core | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_ehca | |
| + local mod=ib_ehca | |
| + local unload_log | |
| + is_module ib_ehca | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_ehca | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload iw_nes | |
| + local mod=iw_nes | |
| + local unload_log | |
| + is_module iw_nes | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w iw_nes | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload cxgb3i | |
| + local mod=cxgb3i | |
| + local unload_log | |
| + is_module cxgb3i | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w cxgb3i | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload iw_cxgb3 | |
| + local mod=iw_cxgb3 | |
| + local unload_log | |
| + is_module iw_cxgb3 | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w iw_cxgb3 | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload cxgb3 | |
| + local mod=cxgb3 | |
| + local unload_log | |
| + is_module cxgb3 | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w cxgb3 | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload iw_cxgb4 | |
| + local mod=iw_cxgb4 | |
| + local unload_log | |
| + is_module iw_cxgb4 | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w iw_cxgb4 | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload cxgb4i | |
| + local mod=cxgb4i | |
| + local unload_log | |
| + is_module cxgb4i | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w cxgb4i | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload cxgb4 | |
| + local mod=cxgb4 | |
| + local unload_log | |
| + is_module cxgb4 | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w cxgb4 | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_qib | |
| + local mod=ib_qib | |
| + local unload_log | |
| + is_module ib_qib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_qib | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mana_ib | |
| + local mod=mana_ib | |
| + local unload_log | |
| + is_module mana_ib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mana_ib | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload rnbd_client | |
| + local mod=rnbd_client | |
| + local unload_log | |
| + is_module rnbd_client | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rnbd_client | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload rnbd_server | |
| + local mod=rnbd_server | |
| + local unload_log | |
| + is_module rnbd_server | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rnbd_server | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload eth_ipoib | |
| + local mod=eth_ipoib | |
| + local unload_log | |
| + is_module eth_ipoib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w eth_ipoib | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_ipoib | |
| + local mod=ib_ipoib | |
| + local unload_log | |
| + is_module ib_ipoib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_ipoib | |
| + RC=0 | |
| + return 0 | |
| + case $mod in | |
| + unload_rec ib_ipoib | |
| + local mod=ib_ipoib | |
| + shift | |
| + is_module ib_ipoib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_ipoib | |
| + RC=0 | |
| + return 0 | |
| + /sbin/modprobe -r ib_ipoib | |
| + '[' 0 -ne 0 ']' | |
| + is_module ib_ipoib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_ipoib | |
| + RC=1 | |
| + return 1 | |
| + '[' 0 -ne 0 ']' | |
| + is_module ib_ipoib | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_ipoib | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx4_vnic | |
| + local mod=mlx4_vnic | |
| + local unload_log | |
| + is_module mlx4_vnic | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx4_vnic | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_madeye | |
| + local mod=ib_madeye | |
| + local unload_log | |
| + is_module ib_madeye | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_madeye | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_rds | |
| + local mod=ib_rds | |
| + local unload_log | |
| + is_module ib_rds | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_rds | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload hns_roce | |
| + local mod=hns_roce | |
| + local unload_log | |
| + is_module hns_roce | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w hns_roce | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload rds_rdma | |
| + local mod=rds_rdma | |
| + local unload_log | |
| + is_module rds_rdma | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rds_rdma | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload rds_tcp | |
| + local mod=rds_tcp | |
| + local unload_log | |
| + is_module rds_tcp | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rds_tcp | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload rds | |
| + local mod=rds | |
| + local unload_log | |
| + is_module rds | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rds | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_ucm | |
| + local mod=ib_ucm | |
| + local unload_log | |
| + is_module ib_ucm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_ucm | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload kdapl | |
| + local mod=kdapl | |
| + local unload_log | |
| + is_module kdapl | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w kdapl | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_srp_target | |
| + local mod=ib_srp_target | |
| + local unload_log | |
| + is_module ib_srp_target | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_srp_target | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload scsi_target | |
| + local mod=scsi_target | |
| + local unload_log | |
| + is_module scsi_target | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w scsi_target | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_srp | |
| + local mod=ib_srp | |
| + local unload_log | |
| + is_module ib_srp | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_srp | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_iser | |
| + local mod=ib_iser | |
| + local unload_log | |
| + is_module ib_iser | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_iser | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_sdp | |
| + local mod=ib_sdp | |
| + local unload_log | |
| + is_module ib_sdp | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_sdp | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload rdma_ucm | |
| + local mod=rdma_ucm | |
| + local unload_log | |
| + is_module rdma_ucm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rdma_ucm | |
| + RC=0 | |
| + return 0 | |
| + case $mod in | |
| + unload_rec rdma_ucm | |
| + local mod=rdma_ucm | |
| + shift | |
| + is_module rdma_ucm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rdma_ucm | |
| + RC=0 | |
| + return 0 | |
| + /sbin/modprobe -r rdma_ucm | |
| + '[' 0 -ne 0 ']' | |
| + is_module rdma_ucm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rdma_ucm | |
| + RC=1 | |
| + return 1 | |
| + '[' 0 -ne 0 ']' | |
| + is_module rdma_ucm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rdma_ucm | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload rdma_cm | |
| + local mod=rdma_cm | |
| + local unload_log | |
| + is_module rdma_cm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w rdma_cm | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload iw_cm | |
| + local mod=iw_cm | |
| + local unload_log | |
| + is_module iw_cm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w iw_cm | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_cm | |
| + local mod=ib_cm | |
| + local unload_log | |
| + is_module ib_cm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_cm | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_local_sa | |
| + local mod=ib_local_sa | |
| + local unload_log | |
| + is_module ib_local_sa | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_local_sa | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload findex | |
| + local mod=findex | |
| + local unload_log | |
| + is_module findex | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w findex | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload auxiliary | |
| + local mod=auxiliary | |
| + local unload_log | |
| + is_module auxiliary | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w auxiliary | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlxdevm | |
| + local mod=mlxdevm | |
| + local unload_log | |
| + is_module mlxdevm | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlxdevm | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx5_vdpa | |
| + local mod=mlx5_vdpa | |
| + local unload_log | |
| + is_module mlx5_vdpa | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_vdpa | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx5_vfio_pci | |
| + local mod=mlx5_vfio_pci | |
| + local unload_log | |
| + is_module mlx5_vfio_pci | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_vfio_pci | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx5_fwctl | |
| + local mod=mlx5_fwctl | |
| + local unload_log | |
| + is_module mlx5_fwctl | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_fwctl | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload fwctl | |
| + local mod=fwctl | |
| + local unload_log | |
| + is_module fwctl | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w fwctl | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload mlx5_dpll | |
| + local mod=mlx5_dpll | |
| + local unload_log | |
| + is_module mlx5_dpll | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w mlx5_dpll | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_sa | |
| + local mod=ib_sa | |
| + local unload_log | |
| + is_module ib_sa | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_sa | |
| + RC=1 | |
| + return 1 | |
| + for mod in $UNLOAD_MODULES | |
| + unload ib_uverbs | |
| + local mod=ib_uverbs | |
| + local unload_log | |
| + is_module ib_uverbs | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_uverbs | |
| + RC=0 | |
| + return 0 | |
| + case $mod in | |
| + unload_rec ib_uverbs | |
| + local mod=ib_uverbs | |
| + shift | |
| + is_module ib_uverbs | |
| + local RC | |
| + /sbin/lsmod | |
| + grep -w ib_uverbs | |
| + RC=0 | |
| + return 0 | |
| + /sbin/modprobe -r ib_uverbs | |
| + '[' 1 -ne 0 ']' | |
| ++ /sbin/rmmod ib_uverbs | |
| ++ grep 'is in use by' | |
| ++ sed -r -e 's/.*use by[:]* //g' | |
| ++ sed -e 's/,/ /g' | |
| + for dep in `/sbin/rmmod $mod 2>&1 | grep "is in use by" | sed -r -e 's/.*use by[:]* //g' | sed -e 's/,/ /g'` | |
| ++ echo ib_mthca mlx5_fpga_tools mlx5_ib mlx5_core mlx4_ib ib_ipath ipath_core ib_ehca iw_nes cxgb3i iw_cxgb3 cxgb3 iw_cxgb4 cxgb4i cxgb4 ib_qib mana_ib rnbd_client rnbd_server eth_ipoib ib_ipoib mlx4_vnic ib_madeye ib_rds hns_roce rds_rdma rds_tcp rds ib_ucm kdapl ib_srp_target scsi_target ib_srp ib_iser ib_sdp rdma_ucm rdma_cm iw_cm ib_cm ib_local_sa findex auxiliary mlxdevm mlx5_vdpa mlx5_vfio_pci mlx5_fwctl fwctl mlx5_dpll ib_sa ib_uverbs ib_umad ib_mad ib_core ib_addr ib_netlink rdma_rxe mlxfw vfio_mdev scsi_transport_srp | |
| ++ grep -q nvidia_peermem | |
| + '[' 0 -eq 0 ']' | |
| + '[' Xmanual '!=' Xauto ']' | |
| + rm_mod ib_uverbs | |
| + local mod=ib_uverbs | |
| + shift | |
| ++ /sbin/rmmod ib_uverbs | |
| + unload_log='rmmod: ERROR: Module ib_uverbs is in use by: nvidia_peermem' | |
| + '[' 1 -ne 0 ']' | |
| + echo_failure 'Unloading ib_uverbs' | |
| + echo -n Unloading ib_uverbs | |
| Unloading ib_uverbs+ '[' color = color ']' | |
| + echo -en '\033[60G' | |
| [60G+ echo -n '[' | |
| [+ '[' color = color ']' | |
| + echo -en '\033[1;31m' | |
| [1;31m+ echo -n FAILED | |
| FAILED+ '[' color = color ']' | |
| + echo -en '\033[0;39m' | |
| [0;39m+ echo -n ']' | |
| ]+ echo -e '\r' | |
| + return 1 | |
| + '[' '!' -z 'rmmod: ERROR: Module ib_uverbs is in use by: nvidia_peermem' ']' | |
| + echo rmmod: ERROR: Module ib_uverbs is in use by: nvidia_peermem | |
| rmmod: ERROR: Module ib_uverbs is in use by: nvidia_peermem | |
| + '[' '!' -z ']' | |
| + cleanup | |
| + /bin/rm -f /var/run/mlx_os_booting | |
| + exit 1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment