Lossless RoCE Configuration for MLNX-OS Switches and Linux Drivers in DSCP-Based PFC and DSCP-Based ECN

configuration.txt

Swtich:
switch-mlnxos [standalone: master] (config) # pool ePool0 direction egress-mc size 4194304 type dynamic
switch-mlnxos [standalone: master] (config) # pool ePool1 direction egress size 16777000 type dynamic
switch-mlnxos [standalone: master] (config) # pool iPool0 direction ingress size 4194304 type dynamic
switch-mlnxos [standalone: master] (config) # pool iPool1 direction ingress size 4194304 type dynamic
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10 qos trust L3 
switch-mlnxos [standalone: master] (config) # interface ethernet 1/11 qos trust L3 
switch-mlnxos [standalone: master] (config) # interface ethernet 1/12 qos trust L3
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg3 bind switch-priority 3
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg6 bind switch-priority 6
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg0 map pool iPool0 type lossy reserved 20480 shared alpha 8
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg3 map pool iPool1 type lossless reserved 70K xoff 17000 xon 17000 shared alpha inf
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg6 map pool iPool1 type lossy reserved 20480 shared alpha 8
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 egress-buffer ePort.tc0 map pool ePool0 reserved 1500 shared alpha 2
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 egress-buffer ePort.tc3 map pool ePool1 reserved 1500 shared alpha inf
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 egress-buffer ePort.tc6 map pool ePool1 reserved 1500 shared alpha 2
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 0 bind switch-priority 0
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 3 bind switch-priority 3
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 6 bind switch-priority 6
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 6 dcb ets strict
switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 3 congestion-control ecn minimum-absolute 5 maximum-absolute 40

NIC:
# mlnx_qos -i ens6f0 --trust dscp
# echo 106 > /sys/class/infiniband/mlx5_0/tc/1/traffic_class
# cma_roce_tos -d mlx5_0 -t 106
# sysctl -w net.ipv4.tcp_ecn=1
# mlnx_qos -i ens6f0 --pfc 0,0,0,1,0,0,0,0

on NIC side, TOS 104 is also acceptable

Now the configuration on modern Onyx (e.g. 3.7.1134) is simpler.

   traffic pool roce type lossless
   traffic pool roce memory percent 50.00
   traffic pool roce map switch-priority 3

   interface ethernet 1/1-1/16 qos trust L3
   interface ethernet 1/1-1/16 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500

   interface ethernet 1/1-1/16 traffic-class 6 dcb ets strict