Notes about results of ibv_generate_infiniband_test_load.
Currently, for RoCE hard coded to use RoCEv1 only.
Dual 10 GbE ports connected to a switch. AlmaLinux release 8.10 with Kernel 4.18.0-553.104.1.el8_10.x86_64.
Initially the active MTU set to 1024:
$ ibv_devinfo
hca_id: rocep33s0f0
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 9803:9b03:0077:e152
sys_image_guid: 9803:9b03:0077:e152
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110004
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
hca_id: rocep33s0f1
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 9803:9b03:0077:e153
sys_image_guid: 9803:9b03:0077:e152
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110004
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
Results:
[mr_halfword@skylake-alma release]$ ibv_generate_infiniband_test_load/ibv_generate_infiniband_test_load 0
PRBS32 pattern period is 4294967295
Press Ctrl-C to stop the RDMA test load
^C
rocep33s0f1 port 1 rx_buffer compare : PASS
rocep33s0f0 port 1 -> rocep33s0f1 port 1 RDMA write transmitted 525059751936 bytes in 462.273715 seconds, 1135.8 Mbytes/sec
rocep33s0f0 port 1 transmitted 563887357956 bytes in 462.273715 seconds, 1219.8 Mbytes/sec
rocep33s0f0 port 1 received 564025459276 bytes in 462.273715 seconds, 1220.1 Mbytes/sec
rocep33s0f0 port 1 rx_buffer compare : PASS
rocep33s0f1 port 1 -> rocep33s0f0 port 1 RDMA write transmitted 525059751936 bytes in 462.241296 seconds, 1135.9 Mbytes/sec
rocep33s0f1 port 1 transmitted 564025556140 bytes in 462.241296 seconds, 1220.2 Mbytes/sec
rocep33s0f1 port 1 received 563887357956 bytes in 462.241296 seconds, 1219.9 Mbytes/sec
Set the Ethernet device MTU to 9600 bytes:
[mr_halfword@skylake-alma release]$ sudo ip link set ens1f0 mtu 9600
[sudo] password for mr_halfword:
[mr_halfword@skylake-alma release]$ sudo ip link set ens1f1 mtu 9600
RoCE active MTU then increases to 4096:
$ ibv_devinfo
hca_id: rocep33s0f0
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 9803:9b03:0077:e152
sys_image_guid: 9803:9b03:0077:e152
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110004
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
hca_id: rocep33s0f1
transport: InfiniBand (0)
fw_ver: 14.32.1010
node_guid: 9803:9b03:0077:e153
sys_image_guid: 9803:9b03:0077:e152
vendor_id: 0x02c9
vendor_part_id: 4117
hw_ver: 0x0
board_id: MT_2420110004
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
Results:
[mr_halfword@skylake-alma release]$ ibv_generate_infiniband_test_load/ibv_generate_infiniband_test_load 0
PRBS32 pattern period is 4294967295
Press Ctrl-C to stop the RDMA test load
^C
rocep33s0f1 port 1 rx_buffer compare : PASS
rocep33s0f0 port 1 -> rocep33s0f1 port 1 RDMA write transmitted 6019128229888 bytes in 4983.480160 seconds, 1207.8 Mbytes/sec
rocep33s0f0 port 1 transmitted 6176364832464 bytes in 4983.480160 seconds, 1239.4 Mbytes/sec
rocep33s0f0 port 1 received 6176369936084 bytes in 4983.480160 seconds, 1239.4 Mbytes/sec
rocep33s0f0 port 1 rx_buffer compare : PASS
rocep33s0f1 port 1 -> rocep33s0f0 port 1 RDMA write transmitted 6019128229888 bytes in 4983.495318 seconds, 1207.8 Mbytes/sec
rocep33s0f1 port 1 transmitted 6176369977964 bytes in 4983.495318 seconds, 1239.4 Mbytes/sec
rocep33s0f1 port 1 received 6176364832464 bytes in 4983.495318 seconds, 1239.4 Mbytes/sec