Skip to content

Instantly share code, notes, and snippets.

@githubfoam

githubfoam/RDMA Cheat Sheet

Last active November 16, 2022 20:01
Show Gist options
  • Save githubfoam/2d591bda14194b1f162750412242d200 to your computer and use it in GitHub Desktop.
Save githubfoam/2d591bda14194b1f162750412242d200 to your computer and use it in GitHub Desktop.
Download ZIP
RDMA Cheat Sheet
Raw
RDMA Cheat Sheet
RDMA verification
This work assumes there are two nodes to run iscsi target/initiator environment.
--------------------------------------------------------------------------------------------------------
Load the RDMA drivers using the following command line
/etc/init.d/openibd start
If one is using the InfiniBand transport and he doesn't have a managed switch in the subnet, he has to start the Subnet Manager (SM). Doing this in one of the machines in the subnet is enough, this can be done with the following command line
/etc/init.d/opensmd start
--------------------------------------------------------------------------------------------------------
Verify that traffic is working
udaddy script
This script covers RDMA_CM UD connections.
(It establishes a set of unreliable RDMA datagram communication paths between two nodes using the librdmacm, optionally transfers datagrams between the nodes, then tears down the communication)
"return status=0" means good exit (RDMA is running).
Firstly on target server
$ udaddy
udaddy: starting server
receiving data transfers
sending replies
data transfers complete
test complete
return status 0
Secondly on initiator server
$ udaddy -s IP_TARGET_SERVER
udaddy: starting client
udaddy: connecting
initiating data transfers
receiving data transfers
data transfers complete
test complete
return status 0
rdma_server, rdma_client commands
simple RDMA CM connection and ping-pong test (It uses synchronous librdmam calls to establish an RDMA connections between two nodes).
"rdma_client: end 0" means good exit (RDMA is running).
Firstly on target server
$ rdma_server
rdma_server: start
rdma_server: end 0
$ rdma_client -s IP_TARGET_SERVER
rdma_client: start
rdma_client: end 0
Use rdma_cm test on the server
# ucmatose
cmatose: starting server
initiating data transfers
completing sends
receiving data transfers
data transfers complete
cmatose: disconnecting
disconnected
test complete
return status 0
Use rdma_cm test on the client
# ucmatose -s 20.4.3.219
cmatose: starting client
cmatose: connecting
receiving data transfers
sending replies
data transfers complete
test complete
return status 0
server IP:192.168.2.106
on server
# ibv_rc_pingpong -g 0 -d mlx4_0 -i 1
local address: LID 0x0003, QPN 0xb5de9e, PSN 0x9d7046, GID fe80::c:2963:2d42:401
remote address: LID 0x0003, QPN 0xb5de9f, PSN 0xfeec26, GID fe80::c:2963:2d42:401
8192000 bytes in 0.27 seconds = 239.96 Mbit/sec
1000 iters in 0.27 seconds = 273.11 usec/iter
on client
# ibv_rc_pingpong -g 0 -d mlx4_0 -i 2 192.168.2.106
local address: LID 0x0003, QPN 0xb5de9f, PSN 0xfeec26, GID fe80::c:2963:2d42:401
remote address: LID 0x0003, QPN 0xb5de9e, PSN 0x9d7046, GID fe80::c:2963:2d42:401
8192000 bytes in 0.27 seconds = 245.91 Mbit/sec
1000 iters in 0.27 seconds = 266.50 usec/iter
server IP:192.168.11.1
on server
rping -s -a 192.168.11.1 -v
on client
# rping -c -a 192.168.11.1 -v
--------------------------------------------------------------------------------------------------------
Verify that RDMA kernel part is loaded
1-Verify that RDMA kernel part is loaded using service file; Inbox RedHat 6.* installation:
# /etc/init.d/rdma status
Low level hardware support loaded:
mlx4_ib
Upper layer protocol modules:
ib_ipoib
User space access modules:
rdma_ucm ib_ucm ib_uverbs ib_umad
Connection management modules:
rdma_cm ib_cm iw_cm
Configured IPoIB interfaces: none
Currently active IPoIB interfaces: ib0 ib1
2-Verify that RDMA kernel part is loaded using lsmod;In all Linux distributions, lsmod can show the loaded kernel modules
[root@localhost] # lsmod | grep ib
mlx4_ib 113239 0
mlx4_core 189003 2 mlx4_ib,mlx4_en
ib_ipoib 68315 0
ib_ucm 9597 0
ib_uverbs 30216 2 rdma_ucm,ib_ucm
ib_umad 8931 4
ib_cm 30987 3 ib_ipoib,ib_ucm,rdma_cm
ib_addr 5176 2 rdma_ucm,rdma_cm
ib_sa 19056 5 mlx4_ib,ib_ipoib,rdma_ucm,rdma_cm,ib_cm
ib_mad 32968 4 mlx4_ib,ib_umad,ib_cm,ib_sa
ib_core 59893 11 mlx4_ib,ib_ipoib,rdma_ucm,ib_ucm,ib_uverbs,ib_umad,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
--------------------------------------------------------------------------------------------------------
Verify that userspace applications are working
1-Verify that RDMA devices are available
# ibv_devices
device node GUID
------ ----------------
mlx4_0 000c29632d420400
2-Verify that RDMA devices can be accessed
# ibv_devinfo -d mlx4_0
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 1.2.005
node_guid: 000c:2963:2d42:0300
sys_image_guid: 000c:2963:2d42:0200
vendor_id: 0x02c9
vendor_part_id: 25418
hw_ver: 0xa
phys_port_cnt: 2
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 1
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_INIT (2)
max_mtu: 4096 (5)
active_mtu: 256 (1)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
--------------------------------------------------------------------------------------------------------
Run the ibdev2netdev utility to display all associations between the Ethernet devices and the IB devices/ports
# ibdev2netdev
mlx5_0 port 1 ==> ens13f0 (Up)
mlx5_1 port 1 ==> ens13f1 (Down)
--------------------------------------------------------------------------------------------------------
Interface name is ens785f0
ethtool --set-priv-flags ens785f0 sniffer on
run tcpdump to capture the packet
tcpdump -i ens785f0 -XXvv > ~/rdma_traffic.txt
To view the file parsed in wireshark
# tcpdump -i ens785f0 -s 65535 -w rdma_traffic.pcap
on server, server IP:99.99.99.5
# ib_send_bw
on client
# ib_send_bw 99.99.99.5 --report_gbits -F
UDP port 4791 is for RoCEv2 traffic
Download Wireshark with RoCEv2 Support
https://community.mellanox.com/docs/DOC-2362#comment-2604
--------------------------------------------------------------------------------------------------------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment