Toward a method of testing Gluster RDMA with regular ethernet NICs, for the hardware-challenged. For now, I have the installing soft-iwarp on Fedora 18 part down (I think). Up next, the getting it to work with Gluster RDMA part...

soft-iwarp.adoc

Testing Gluster RDMA with Soft-iWARP

There’s a Gluster 3.4 RDMA test day right around the corner, and I want to join in on the fun. The trouble is, I don’t have any RDMA-capable hardware in my lab right now. Undaunted, I hit the Web in search of a software-based solution, one that would at least allow me to run through the tests.

I found a pair of promising-looking options:

• Soft-iWARP

• Soft ROCE

The information out on the web about these projects is a bit thinner than I’d like, but I found a blog post howto on installing Soft-iWARP on Ubuntu 10.10 and another for Debian 6 and figured I’d try it out on Fedora 18.

Here are the steps I followed:

I started with a VM running the x86_64 version of Fedora 18 with all updates applied.

I installed a bunch of dependencies:

yum install -y kernel-devel libtool autoconf rdma  \
libibverbs-utils libibcommon libibcm-devel libibverbs \
git wget yum-utils librdmacm-utils

yum-builddep -y kernel-devel

Next, grab the source for Soft-iWARP:

git clone http://git.gitorious.org/softiwarp/kernel.git

git clone http://git.gitorious.org/softiwarp/userlib.git

Time to build & install:

cd kernel/softiwarp/

make

sudo make install

cd ../../userlib/libsiw-0.1/

./autogen.sh

./autogen.sh (yes, you have to run it twice)

./configure prefix=/usr libdir=/usr/lib64

make

sudo make install

cp siw.driver /etc/libibverbs.d/

There’s a series of modules that need loading, I created a shell script, as suggested in one of the howtos, to load them:

cd ~

vi load-em-up.sh

#!/bin/bash
modprobe ib_addr
modprobe ib_cm
modprobe ib_core
modprobe ib_mad
modprobe ib_sa
modprobe ib_ucm
modprobe ib_umad
modprobe ib_uverbs
modprobe iw_cm
modprobe rdma_cm
modprobe rdma_ucm
modprobe siw

sh load-em-up.sh

From here, you can try the commands ibv_devinfo and ibv_devices to get info on your interface(s) — on my one-NIC VM, interfaces appeared for both my virtual NIC and my loopback interface.

You can run the command "rping -s" paired with "rping -c -a 127.0.0.1 -v" on another shell, to ping your loopback interface to see if everything’s working.

Swap out the "127.0.0.1" with the IP address of your test machine to try pinging the non-loopback interface. Use Ctrl-C to break the cycle-of-pingage.

Very interesting! Could you share the result you find?