Skip to content

Instantly share code, notes, and snippets.

@ksonbol

ksonbol/Installing SoftiWARP on Ubuntu 16.04.md

Last active May 17, 2021 09:44
Show Gist options
  • Save ksonbol/d09727079501832b4a9608eaee42f4e9 to your computer and use it in GitHub Desktop.
Save ksonbol/d09727079501832b4a9608eaee42f4e9 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Installing SoftiWARP on Ubuntu 16.04.md

Instructions for installing SoftiWARP and common issues encountered, tested with Ubuntu 16.04.1 with Kernel 4.15.0-45-generic.

- Setting up the Linux kernel dev environment:

 sudo apt-get install fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge
 sudo apt-get build-dep linux
 sudo apt-get install git-core libncurses5 libncurses5-dev libelf-dev binutils-dev

- Setting up the Infiniband / libibverbs environment

sudo apt-get install libibverbs1 libibcm1 libibcm-dev  libibverbs-dev  libibcommon1 ibverbs-utils

check if “/etc/udev/rules.d/40-ib.rules” exist: If the file doesn’t exist , create it and populate it with (you need to reboot after):

 ####  /etc/udev/rules.d/40-ib.rules  ####
 KERNEL=="umad*", NAME="infiniband/%k"
 KERNEL=="issm*", NAME="infiniband/%k"
 KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
 KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
 KERNEL=="uat", NAME="infiniband/%k", MODE="0666"
 KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
 KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"
 ########

- Verifying the Infiniband / libibverbs environment Run these command in the following order:

 sudo modprobe  rdma_cm
 sudo modprobe ib_uverbs
 sudo modprobe rdma_ucm
 sudo lsmod

Verify rdma modules exist in output of last command. The modules are: rdma_ucm, ib_uverbs, rdma_cm, ib_cm, iw_cm, ib_sa, ib_mad, ib_core, ib_addr.

 ls /dev/infiniband/

You should see rdma_cm

Before building the SoftiWARP library, let's download libibverbs first:

To avoid such an error:

 configure: error: ibv_get_device_list() not found.  libsiw requires libibverbs.

Instead of downloading libibverbs with apt-get,

cd libibverbs-1.2.1
./configure
Make
sudo make install
sudo cp examples/ibv_* /usr/bin/

This last line puts commands like ibv_devices in their right places (replaces the ones the ones installed with apt-get if any, source: zrlio/softiwarp#12 (comment)).

- Getting the compile / dev environment for Soft-iWARP:

sudo apt-get install libtool autoconf

- Installing Soft-iWARP:

Download the development branch from here: https://github.com/zrlio/softiwarp/tree/dev-siw.mem_ext

  • Compiling and installing the Soft-iWARP kernel module: Inside the soft-iwarp kernel module folder, run:
make
sudo make install
  • Compiling and installing the Soft-iWARP userlib: Inside the soft-iwarp userlib folder:
./autogen.sh
Run AGAIN  ./autogen.sh
./configure
make
sudo make install
  • At this stage soft-iwarp installs the infiniband stuff in “/usr/local/etc/libibverbs.d”. We need to make a symbolic link at /etc/infiniband.d:
sudo ln -s /usr/local/etc/libibverbs.d /etc//libibverbs.d

note: the // is not a missepll.

  • Inserting the Soft-iWARP kernel module:
sudo modprobe  rdma_cm
sudo modprobe ib_uverbs
sudo modprobe rdma_ucm
sudo insmod  /lib/modules/$(uname -r)/extra/siw.ko
sudo lsmod

From last command, check if all the modules are correctly installed (as we did earlier).

ls /dev/infiniband/

You should see at least rdma_cm and uverbs0.

We also run sudo ldconfig here to avoid such errors as:

 ibv_devinfo: error while loading shared libraries: libibverbs.so.1: cannot open shared object file: No such file or directory

- Verifying the Soft-iWARP is correctly installed:

Run these and check you get two entries for each command:

 ibv_devinfo
 ibv_devices

You should see the device information in both commands.

Source: Most steps are taken from this post: https://www.reflectionsofthevoid.com/2011/03/how-to-install-soft-iwarp-on-ubuntu.html

IT WORKS? GREAT, now let's load the modules automatically at startup:

  • First, create a symbolic link to siw.ko so modprobe can load it:

    sudo ln -s /lib/modules/$(uname -r)/extra/siw.ko /lib/modules/$(uname -r)/
sudo depmod -a

  • Next, we will create a systemd service that loads softiwarp required modules at "start", and unloads them at "stop". We will create two files:

  1. Create the script

sudo nano /usr/bin/softiwarp

Fill file with these contents:

#!/bin/sh

 start() {
   modprobe rdma_cm
   modprobe ib_uverbs
   modprobe rdma_ucm
   modprobe siw
 }

 stop() {
   modprobe -r rdma_cm
   modprobe -r ib_uverbs
   modprobe -r rdma_ucm
   modprobe -r siw
 }

 case $1 in
   start|stop) "$1" ;;
 esac

Make sure there is no space before the first line!

Then, add exec permission to the file

 sudo chmod +x /usr/bin/softiwarp
  1. Create the service file

sudo nano /etc/systemd/system/softiwarp.service

Fill file with the following:

 [Unit]
 Description=Load RDMA and SoftiWARP modules

 [Service]
 Type=oneshot
 ExecStart=/usr/bin/softiwarp start
 ExecStop=/usr/bin/softiwarp stop
 RemainAfterExit=yes

 [Install]
 WantedBy=multi-user.target

Add exec permission to the service file (this may not be required)

 sudo chmod +x /etc/systemd/system/softiwarp.service
  1. Run sudo systemctl enable softiwarp.service
  2. Restart, then run sudo service softiwarp restart.
  3. Everything should work now! check that you get two entries (one is for loopback) when running: ibv_devices.

Note: On startup, for some reason this attaches the modules to the loopback address only, so you need to run

 sudo service softiwarp restart

after each boot to use softiwarp, at least you run one command only now!

Common issues

  • When running sudo make installl for the kernel you see these errors
SSL error:02001002:system library:fopen:No such file or directory: ../crypto/bio/bss_file.c:74
SSL error:2006D080:BIO routines:BIO_new_file:no such file: ../crypto/bio/bss_file.c:81

Just ignore them! The build has probably completed successfully, these are just warnings.

  • You get this error: insmod: error inserting 'siw.ko': -1 Invalid module format.

when running: sudo insmod /lib/modules/$(uname -r)/extra/siw.ko

It means that the kernel from which you have built your kernel module is differnt from the one to which you are trying to insert the module. Probably you built it at some time, then later with updates you have a newer kernel version that you are trying to insert the module into.

You can either:

change this to: sudo insmod /lib/modules/old_kernel_version/extra/siw.ko replacing old_kernel_version with the version you built the modules with.

OR

run sudo make clean, then rebuild the kernel module following the above steps.

Inserting the module should work now.

  • When running ibv_devinfo or ibv_devices, you get the error:
libibverbs: Warning: couldn't load driver 'siw': /usr/local/lib/libibverbs.so.1: version 'IBVERBS_PRIVATE_17' not found
(required by /usr/local/lib/libsiw-rdmav2.so)
No IB devices found

Couldn't fix this error yet (this happens with Ubuntu 18.04).

@474172261
Copy link

Hi, I can't locate libibcommon1 in Ubuntu 16.04, can you find it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment