Last active
August 7, 2019 05:18
-
-
Save crazyboycjr/001c9f69e43970ceca12c867884d2b1b to your computer and use it in GitHub Desktop.
Lossless RoCE Configuration for MLNX-OS Switches and Linux Drivers in DSCP-Based PFC and DSCP-Based ECN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Swtich: | |
| switch-mlnxos [standalone: master] (config) # pool ePool0 direction egress-mc size 4194304 type dynamic | |
| switch-mlnxos [standalone: master] (config) # pool ePool1 direction egress size 16777000 type dynamic | |
| switch-mlnxos [standalone: master] (config) # pool iPool0 direction ingress size 4194304 type dynamic | |
| switch-mlnxos [standalone: master] (config) # pool iPool1 direction ingress size 4194304 type dynamic | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10 qos trust L3 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/11 qos trust L3 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/12 qos trust L3 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg3 bind switch-priority 3 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg6 bind switch-priority 6 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg0 map pool iPool0 type lossy reserved 20480 shared alpha 8 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg3 map pool iPool1 type lossless reserved 70K xoff 17000 xon 17000 shared alpha inf | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 ingress-buffer iPort.pg6 map pool iPool1 type lossy reserved 20480 shared alpha 8 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 egress-buffer ePort.tc0 map pool ePool0 reserved 1500 shared alpha 2 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 egress-buffer ePort.tc3 map pool ePool1 reserved 1500 shared alpha inf | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 egress-buffer ePort.tc6 map pool ePool1 reserved 1500 shared alpha 2 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 0 bind switch-priority 0 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 3 bind switch-priority 3 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 6 bind switch-priority 6 | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 6 dcb ets strict | |
| switch-mlnxos [standalone: master] (config) # interface ethernet 1/10-1/13 traffic-class 3 congestion-control ecn minimum-absolute 5 maximum-absolute 40 | |
| NIC: | |
| # mlnx_qos -i ens6f0 --trust dscp | |
| # echo 106 > /sys/class/infiniband/mlx5_0/tc/1/traffic_class | |
| # cma_roce_tos -d mlx5_0 -t 106 | |
| # sysctl -w net.ipv4.tcp_ecn=1 | |
| # mlnx_qos -i ens6f0 --pfc 0,0,0,1,0,0,0,0 |
Now the configuration on modern Onyx (e.g. 3.7.1134) is simpler.
traffic pool roce type lossless
traffic pool roce memory percent 50.00
traffic pool roce map switch-priority 3
interface ethernet 1/1-1/16 qos trust L3
interface ethernet 1/1-1/16 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500
interface ethernet 1/1-1/16 traffic-class 6 dcb ets strict
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
on NIC side, TOS 104 is also acceptable