Thunderbolt Networking Setup

proxmox-tb-net.md

Thunderbolt Networking

this gist is part of this series

NOTE FOR THIS TO BE RELIABLE ON NODE RESTARTS YOU WILL NEED PROXMOX KERNEL 6.2.16-14-pve OR HIGER

This fixes issues i bugged with the thunderbolt / thunderbolt-net maintainers (i will take everyones thanks now, lol)

Install LLDP - this is great to see what nodes can see which.

• install lldpctl with apt install lldpd

Load Kernel Modules

• add thunderbolt and thunderbolt-net kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-)
  1. nano /etc/modules add modules at bottom of file, one on each line
  2. save using x then y then enter

Prepare /etc/network/interfaces

doing this means we don't have to give each thunderbolt a manual IPv6 addrees and that these addresses stay constant no matter what Add the following to each node using nano /etc/network/interfaces

If you see any sections called thunderbolt0 or thunderbol1 delete them at this point.

Create entries to prepopulate gui with reminder

Doing this means we don't have to give each thunderbolt a manual IPv6 or IPv4 addrees and that these addresses stay constant no matter what.

Add the following to each node using nano /etc/network/interfaces this to remind you not to edit en05 and en06 in the GUI

This fragment should go between the existing auto lo section and adapater sections.

iface en05 inet manual
#do not edit it GUI

iface en06 inet manual
#do not edit in GUI

If you see any thunderbol sections delete them from the file before you save it.

*DO NOT DELETE the source /etc/network/interfaces.d/* this will always exist on the latest versions and should be the last or next to last line in /interfaces file

Rename Thunderbolt Connections

This is needed as proxmox doesn't recognize the thunderbolt interface name. There are various methods to do this. This method was selected after trial and error because:

• the thunderboltX naming is not fixed to a port (it seems to be based on sequence you plug the cables in)
• the MAC address of the interfaces changes with most cable insertion and removale events

1. use udevadm monitor command to find your device IDs when you insert and remove each TB4 cable. Yes you can use other ways to do this, i recommend this one as it is great way to understand what udev does - the command proved more useful to me than the syslog or lspci command for troublehsooting thunderbolt issues and behavious. In my case my two pci paths are 0000:00:0d.2and 0000:00:0d.3 if you bought the same hardware this will be the same on all 3 units. Don't assume your PCI device paths will be the same as mine.

2. create a link file using nano /etc/systemd/network/00-thunderbolt0.link and enter the following content:

[Match]
Path=pci-0000:00:0d.2
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05

3. create a second link file using nano /etc/systemd/network/00-thunderbolt1.link and enter the following content:

[Match]
Path=pci-0000:00:0d.3
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en06

Set Interfaces to UP on reboots and cable insertions

This section en sure that the interfaces will be brought up at boot or cable insertion with whatever settings are in /etc/network/interfaces - this shouldn't need to be done, it seems like a bug in the way thunderbolt networking is handled (i assume this is debian wide but haven't checked).

1. create a udev rule to detect for cable insertion using nano /etc/udev/rules.d/10-tb-en.rules with the following content:

ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-en06.sh"

2. save the file

3. create the first script referenced above using nano /usr/local/bin/pve-en05.sh and with the follwing content:

#!/bin/bash

# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
/usr/sbin/ifup en05

save the file and then

3. create the second script referenced above using nano /usr/local/bin/pve-en06.sh and with the follwing content:

#!/bin/bash

# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
/usr/sbin/ifup en06

and save the file

4. make both scripts executable with chmod +x /usr/local/bin/*.sh
5. run update-initramfs -u -k all to propogate the new link files into initramfs
6. Reboot (restarting networking, init 1 and init 3 are not good enough, so reboot)

Enabling IP Connectivity

proceed to the next gist

Slow thunderbolt perf

if you are having speed issues make sure the following is set on the kernel command line in /etc/default/grub file intel_iommu=on iommu=pt one set be sure to run update-grub and reboot

everyones grub command line is different this is mine because i also have i915 virtualization, if you get this wrong you can break your machine, if you are not doing that you don't need the i915 entries you see below

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" (note if you have more things in your cmd line DO NOT REMOVE them, just add the two intel ones, doesnt matter where.

Extra Debugging for Thunderbolt

dynamic kernel tracing - adds more info to dmesg, doesn't overhwelm dmesg

I have only tried this on 6.8 kernels, so YMMV If you want more TB messages in dmesg to see why connection might be failing here is how to turn on dynamic tracing

For bootime you will need to add it to the kernel command line by adding thunderbolt.dyndbg=+p to your /etc/default/grub file, running update-grub and rebooting.

To expand the example above"

`GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt thunderbolt.dyndbg=+p"`  

Don't forget to run update-grub after saving the change to the grub file.

For runtime debug you can run the following command (it will revert on next boot) so this cant be used to cpature what happens at boot time.

`echo -n 'module thunderbolt =p' > /sys/kernel/debug/dynamic_debug/control`

install tbtools

these tools can be used to inspect your thundebolt system, note they rely on rust to be installedm you must use the rustup script below and not intsall rust by package manager at this time (9/15/24)

apt install pkg-config libudev-dev git curl
curl https://sh.rustup.rs -sSf | sh
git clone https://github.com/intel/tbtools
restart you ssh session
cd tbtools
cargo install --path .

The script doing the job is as follows :

#!/bin/bash

ip link set thunderbolt0 down
ip link set thunderbolt0 name en06
ip link set en06 up

this makes me think your matching rules were not specific, here you are renaming thunderbolt0, thing is which port is thunderbolt0 can and will vary from boot to boot, the PCI matching approach i used means you always know which physical port is en05 and which is en06

the fact you only have to do this one makes me think you have an issue with your original matching rules in some way, infact i strongly suspect you have the wrong dev paths in some way, also you globbing doesn't look right, if you want to think these should work equally as well as what you had (you dont need the /dev and all the man pages show the syntax like this below

path=pci0000:00/0000:00:0d.3/domain1/1-0/1-1
path=pci0000:00/0000:00:0d.3/domain1/1-0/1-3

@fesnault this may be the cause of your issue with name not changing with the link file and maybe some of the other issues we see

"All .link files do is to suggest the "best" name which is ignored if
NAME has already been set. They do not really rename anything
themselves. Besides, OP cannot use .link files because there is no
persistent device property that can be matched by them.

It is possible that there are conflicting rules in initrd. It is also
possible that udev races with the networkd."

https://lists.freedesktop.org/archives/systemd-devel/2024-August/050657.html

this would be, for example if you have thunderbolt0 in one of the network interface files or in interfaces.d you may end up in a race condition... make sure thunderbolt0 isn't define anywhere else

i am starting to think we are just hitting inherent timing and race conditions based on poor designs in systemd and networkd, and that is why everyone is seeing something slightly different

Thank you @nickglott ! I did that, with 0-11 as my 3 nodes are using i9 13900H, then I did if down and ifup on both en05 and en06 interfaces. But the results are the same, 16 Gbit/s and high retry count.

root@ms01-01:~# iperf3 -c fc00::83 -bidir
Connecting to host fc00::83, port 5201
[  5] local fc00::81 port 33400 connected to fc00::83 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.07 GBytes  9.21 Gbits/sec  264   1.37 MBytes       
[  5]   1.00-2.00   sec  2.17 GBytes  18.6 Gbits/sec  677   1.19 MBytes       
[  5]   2.00-3.00   sec  1.62 GBytes  13.9 Gbits/sec  517   1.25 MBytes       
[  5]   3.00-4.00   sec  2.19 GBytes  18.8 Gbits/sec  670   1.19 MBytes       
[  5]   4.00-5.00   sec  2.19 GBytes  18.8 Gbits/sec  655   1.25 MBytes       
[  5]   5.00-6.00   sec  2.19 GBytes  18.8 Gbits/sec  619   1.31 MBytes       
[  5]   6.00-7.00   sec  1.07 GBytes  9.22 Gbits/sec  285   1.62 MBytes       
[  5]   7.00-8.00   sec  2.74 GBytes  23.6 Gbits/sec  677   1.75 MBytes       
[  5]   8.00-9.00   sec  1.03 GBytes  8.87 Gbits/sec  353   2.12 MBytes       
[  5]   9.00-10.00  sec  2.20 GBytes  18.9 Gbits/sec  676   1.31 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  18.5 GBytes  15.9 Gbits/sec  5393             sender
[  5]   0.00-10.00  sec  18.5 GBytes  15.9 Gbits/sec                  receiver

iperf Done.
root@ms01-01:~# iperf3 -c fc00::82 -bidir
Connecting to host fc00::82, port 5201
[  5] local fc00::81 port 49032 connected to fc00::82 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.02 GBytes  17.4 Gbits/sec  454   2.31 MBytes       
[  5]   1.00-2.00   sec  1.37 GBytes  11.8 Gbits/sec  341   2.37 MBytes       
[  5]   2.00-3.00   sec  1.45 GBytes  12.4 Gbits/sec  352   2.00 MBytes       
[  5]   3.00-4.00   sec  2.22 GBytes  19.1 Gbits/sec  529   2.31 MBytes       
[  5]   4.00-5.00   sec  2.20 GBytes  18.9 Gbits/sec  561   1.31 MBytes       
[  5]   5.00-6.00   sec  2.77 GBytes  23.8 Gbits/sec  592   1.37 MBytes       
[  5]   6.00-7.00   sec  1.58 GBytes  13.6 Gbits/sec  362   2.18 MBytes       
[  5]   7.00-8.00   sec  2.19 GBytes  18.8 Gbits/sec  521   1.81 MBytes       
[  5]   8.00-9.00   sec  1.90 GBytes  16.3 Gbits/sec  335   2.00 MBytes       
[  5]   9.00-10.00  sec   968 MBytes  8.12 Gbits/sec  193   2.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  18.7 GBytes  16.0 Gbits/sec  4240             sender
[  5]   0.00-10.00  sec  18.7 GBytes  16.0 Gbits/sec                  receiver

iperf Done.

Maybe I forgot something ?

Hi @fesnault, i am just curious if you ever got this resolved? When using 3 MS-01's I got the same thing above. One node would hit the 20-22GB/S and the other 2 would only hit 15-16. Never found out why it was like that or spent more time fixing but would interested to know if anyone has found a solution.

For me, I kinda just got 3x 100g Mellanox Connect 2 port cards and called it a day. However, the card is PCIE Gen3x16 and the MS01 is PCIE Gen4x8. I did lose 20gbps and only getting 80 instead of the 100. So if anyways willing to dish the money, it works but at that point probably would be wiser to get the 50gb cards or something.

@uvalleza & all: Quick summary from my last few days using 3 um790 pro from minisforum:

• Context: Ubuntu 24.04 LTS on 3 "nuc" with AMD cpu. (I was told: usb-4 doesn't necessary means thunderbolt-3, the spec is a pick & choose; on top of that until recently thunderbolt was a intel only feature)
• The speed i get on direct link is around ~12Gbits like you do
• The frr needs a "reload" after boot (no need to play with the wires, unless the logs shows invalid config for usb port x) see code snipped bellow
• Downgrading the bios/uefi from 1.09 to 1.07 gave me less "invalid config for usb x" kind of errors and more reliability after reboots (i still have to unplug-replug some wires sometimes) also, it seems that my speed went from 10-11gbits to 12-13 gbits, but i can't really confirm. => what bios/uefi version are you running with ?
• I had to stop using encrypted boot drives as it required me to unplug the thunderbolt links to let the hdmi work, then replug it all.
• cables are indeed placed the same way you do
• my frr setup seems to be working, but once I remove 1 of the link, the speed is about 2Mbits (yes mega bits) when it needs to do 1 more hop through the 2 other thunderbolt links => i have yet to figure out that part

To auto-reload the frr configuration after reboot (required otherwise it fails to see the thunderbolt links and I get 3 independent nodes that don't see each other via vtysh -c "show openfabric topology") Requirement: have your interfaces renamed (see "tbt" in script) as explained in the first post by scyto (don't use hyphen in interface names, it wasn't working for me)

#!/bin/sh
# Delayed start script to tell frr to reload ensuring that it sees thunderbolt links towards other nodes.
# condition: is there any tbt network interface and frr service up
COUNTER=0
while [ ${COUNTER} -lt 5 ]; do
	sleep 1;
	TEST=$(ip a | grep ": tbt" | grep "UP" | awk 'BEGIN { ORS=""}; {print $2}')
	if [ ${#TEST} -ge 2 ]; then
		TEST_SVC=$(service frr status | grep "active (running)")
		if [ ${#TEST_SVC} -ge 2 ]; then
			service frr reload;
			echo "frr service reload request sent"
			exit 0;
		fi
	fi
	COUNTER=$((COUNTER+1));
done
echo "Failed to request frr service reload: request NOT sent"
exit 1;

[Unit]
After=network.target

[Service]
ExecStart=/usr/local/bin/restart-frr.sh

[Install]
WantedBy=default.target

Note: The script is called restart, but after some testing, I realised that reload was enough.
To all: thank you for sharing your experience, its a great help & motivation to figure out what's going sideways 😄

Hey, just wanted to thank you for your input first. I happen to have the same setup as you, only with 5 nodes. Would you be so kind to share your config with this script: https://github.com/Allistah/Get-Proxmox-Thunderbolt-Config/blob/main/get-thunderbolt-config.sh (Thank you Allistah)

Your help is greatly appreciated, I am pretty desperate here lol

Here my output: https://privatebin.net/?04649f34b02189de#57hHKfwusnma1Bre1gSkwm2kMi1rucqe8hpG61GSU1cT

Edit:

Also, do I put these two segments into the same .sh file and make it executable, or do I put them at different places?

#!/bin/sh
# Delayed start script to tell frr to reload ensuring that it sees thunderbolt links towards other nodes.
# condition: is there any tbt network interface and frr service up
COUNTER=0
while [ ${COUNTER} -lt 5 ]; do
	sleep 1;
	TEST=$(ip a | grep ": tbt" | grep "UP" | awk 'BEGIN { ORS=""}; {print $2}')
	if [ ${#TEST} -ge 2 ]; then
		TEST_SVC=$(service frr status | grep "active (running)")
		if [ ${#TEST_SVC} -ge 2 ]; then
			service frr reload;
			echo "frr service reload request sent"
			exit 0;
		fi
	fi
	COUNTER=$((COUNTER+1));
done
echo "Failed to request frr service reload: request NOT sent"
exit 1;

[Unit]
After=network.target

[Service]
ExecStart=/usr/local/bin/restart-frr.sh

[Install]
WantedBy=default.target

Will share the config once i set this up with a friend. He just happened to buy my last UM790 Pro to complete his 3 node UM790 Pro cluster and will be setting this up for him. Don't really expect much issues on a 3 node but could kinda see why you might have issues on a 5 node cluster.

No expert on this but here's my two cents. You have 5 nodes.

1. How would each node communicate with itself. In a 3 node, you have 2 ports each allowing each one to connect to one another directly. In a 5 node, you only still have 2 ports, unless you're getting 2 more ports. I don't see the above really working unless you're able to jump from one box to another.

Example:
Node 1 is connected to Node 2 and 3.
Node 2 is connected to 4 and 5.

how do you tell Node 1 to go through Node 2 or 3 to get to Node 4/5?

seems like it'll be a routing config.

Anyhow, I'm not sure what your exactly issue is from the above but since I was prepping to use the guide with a friend, figured i chime in the comments :). Will report the results and will share the configs if you still need them.