Thunderbolt Networking Setup

proxmox-tb-net.md

Thunderbolt Networking

this gist is part of this series

NOTE FOR THIS TO BE RELIABLE ON NODE RESTARTS YOU WILL NEED PROXMOX KERNEL 6.2.16-14-pve OR HIGER

This fixes issues i bugged with the thunderbolt / thunderbolt-net maintainers (i will take everyones thanks now, lol)

Install LLDP - this is great to see what nodes can see which.

• install lldpctl with apt install lldpd

Load Kernel Modules

• add thunderbolt and thunderbolt-net kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-)
  1. nano /etc/modules add modules at bottom of file, one on each line
  2. save using x then y then enter

Prepare /etc/network/interfaces

doing this means we don't have to give each thunderbolt a manual IPv6 addrees and that these addresses stay constant no matter what Add the following to each node using nano /etc/network/interfaces

If you see any sections called thunderbolt0 or thunderbol1 delete them at this point.

Create entries to prepopulate gui with reminder

Doing this means we don't have to give each thunderbolt a manual IPv6 or IPv4 addrees and that these addresses stay constant no matter what.

Add the following to each node using nano /etc/network/interfaces this to remind you not to edit en05 and en06 in the GUI

This fragment should go between the existing auto lo section and adapater sections.

iface en05 inet manual
#do not edit it GUI

iface en06 inet manual
#do not edit in GUI

If you see any thunderbol sections delete them from the file before you save it.

*DO NOT DELETE the source /etc/network/interfaces.d/* this will always exist on the latest versions and should be the last or next to last line in /interfaces file

Rename Thunderbolt Connections

This is needed as proxmox doesn't recognize the thunderbolt interface name. There are various methods to do this. This method was selected after trial and error because:

• the thunderboltX naming is not fixed to a port (it seems to be based on sequence you plug the cables in)
• the MAC address of the interfaces changes with most cable insertion and removale events

1. use udevadm monitor command to find your device IDs when you insert and remove each TB4 cable. Yes you can use other ways to do this, i recommend this one as it is great way to understand what udev does - the command proved more useful to me than the syslog or lspci command for troublehsooting thunderbolt issues and behavious. In my case my two pci paths are 0000:00:0d.2and 0000:00:0d.3 if you bought the same hardware this will be the same on all 3 units. Don't assume your PCI device paths will be the same as mine.

2. create a link file using nano /etc/systemd/network/00-thunderbolt0.link and enter the following content:

[Match]
Path=pci-0000:00:0d.2
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05

3. create a second link file using nano /etc/systemd/network/00-thunderbolt1.link and enter the following content:

[Match]
Path=pci-0000:00:0d.3
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en06

Set Interfaces to UP on reboots and cable insertions

This section en sure that the interfaces will be brought up at boot or cable insertion with whatever settings are in /etc/network/interfaces - this shouldn't need to be done, it seems like a bug in the way thunderbolt networking is handled (i assume this is debian wide but haven't checked).

1. create a udev rule to detect for cable insertion using nano /etc/udev/rules.d/10-tb-en.rules with the following content:

ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-en06.sh"

2. save the file

3. create the first script referenced above using nano /usr/local/bin/pve-en05.sh and with the follwing content:

#!/bin/bash

# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
/usr/sbin/ifup en05

save the file and then

3. create the second script referenced above using nano /usr/local/bin/pve-en06.sh and with the follwing content:

#!/bin/bash

# this brings the renamed interface up and reprocesses any settings in /etc/network/interfaces for the renamed interface
/usr/sbin/ifup en06

and save the file

4. make both scripts executable with chmod +x /usr/local/bin/*.sh
5. run update-initramfs -u -k all to propogate the new link files into initramfs
6. Reboot (restarting networking, init 1 and init 3 are not good enough, so reboot)

Enabling IP Connectivity

proceed to the next gist

Slow thunderbolt perf

if you are having speed issues make sure the following is set on the kernel command line in /etc/default/grub file intel_iommu=on iommu=pt one set be sure to run update-grub and reboot

everyones grub command line is different this is mine because i also have i915 virtualization, if you get this wrong you can break your machine, if you are not doing that you don't need the i915 entries you see below

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" (note if you have more things in your cmd line DO NOT REMOVE them, just add the two intel ones, doesnt matter where.

Extra Debugging for Thunderbolt

dynamic kernel tracing - adds more info to dmesg, doesn't overhwelm dmesg

I have only tried this on 6.8 kernels, so YMMV If you want more TB messages in dmesg to see why connection might be failing here is how to turn on dynamic tracing

For bootime you will need to add it to the kernel command line by adding thunderbolt.dyndbg=+p to your /etc/default/grub file, running update-grub and rebooting.

To expand the example above"

`GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt thunderbolt.dyndbg=+p"`  

Don't forget to run update-grub after saving the change to the grub file.

For runtime debug you can run the following command (it will revert on next boot) so this cant be used to cpature what happens at boot time.

`echo -n 'module thunderbolt =p' > /sys/kernel/debug/dynamic_debug/control`

install tbtools

these tools can be used to inspect your thundebolt system, note they rely on rust to be installedm you must use the rustup script below and not intsall rust by package manager at this time (9/15/24)

apt install pkg-config libudev-dev git curl
curl https://sh.rustup.rs -sSf | sh
git clone https://github.com/intel/tbtools
restart you ssh session
cd tbtools
cargo install --path .

I just have to raise a shout-out to @nickglott for his excellent guide above.

I was first struggling with frr stopping when I rebooted one node, this was because I tried to use dependency for frr to en06 and en06. When one of these became disconnected (because another host went down or cable unplugged) then frr would stop. No dice.

Solution: Move the frr restart to /etc/network/interfaces.d/thunderbolt like this:

allow-hotplug en05
iface en05 inet manual
        mtu 65520

allow-hotplug en06
iface en06 inet manual
        mtu 65520

post-up /usr/bin/systemctl restart frr.service 

Then I sollowed @nickglott guide above for frr settings. One thing I missed from @scyto original gist was this part. I had used the same net address on each host. That failed for sure. Not sure if putting IPs in frr is needed, but at least it really works. My thinking is that it was the net ID problem that tripped my setup.

For my /etc/frr/frr.conf
*Change X in Hostname, IP, and NET Address for each node to the following
** TheCore-01 | 10.0.10.10/32 | fc00::10/128 | 49.0001.1111.1111.1111.00 **
** TheCore-02 | 10.0.10.20/32 | fc00::20/128 | 49.0001.2222.2222.2222.00 **
** TheCore-03 | 10.0.10.30/32 | fc00::30/128 | 49.0001.3333.3333.3333.00 **

After these changes my setup works perfect, even when one node goes down, and comes back up. And it works on ipv4 and ipv6. :)

Hi @scyto and everybody!

I had this issue when rebooting any proxmox node that thunderbolt network interface didn't come up. Finally I figured it out and maybe somebody can profit too.

/usr/local/bin/pve-en05.sh

#!/bin/bash

for i in {1..10}; do
    /usr/sbin/ifup en05 && break
    sleep 3
done

/usr/local/bin/pve-en06.sh

#!/bin/bash

for i in {1..10}; do
    /usr/sbin/ifup en06 && break
    sleep 3
done

When triggered by udev, it runs appropriate script, if "ifup" succeeds (exit code 0), the "break" will stop the loop. If exit code is not 0, it will sleep 3 seconds and do it again (up to 10 times in this case, ultimately lasting for 30s). You can adjust sleep time or number of tries in loop.

FYI @ronindesign @ctroyp

I also had the exact same problem as @ronindesign . Found the following error in my logs: "error: Another instance of this program is already running." when I debug the udev script. This error indicates that ifup is running two times but is allowed only one instance, I guess because it is the boot process. At least a quick fix for me was to change like this:

ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-net.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-net.sh"

And the script:

#!/bin/bash
LOGFILE="/tmp/udev-debug.log"
echo "$(date): pve-net.sh triggered by udev" >> $LOGFILE

#Bring up both en05 and en06 
/usr/sbin/ifup -v en05 en06 >> $LOGFILE 2>&1

This way I also get a log in /tmp at every boot to see that the IFs got up well. Which is the best solution (above or mine) I am open to suggestions. But at least this works well. Before this change, at every boot, en05 came up, en06 did not. This caused strange issues since sometimes it worked (proxmox happy, ceph happy), but at some point when you boot another node you could hit the downed interface and ceph not happy anymore :P

Check in Proxmox that all IFs are up after boot to see if you are exposed to this problem.

Both en05 and en06 should be Active: Yes on all nodes. If they are not (after boot especially) then have a look at this solution.

For completeness I can mention that I moved IP away from frr and back to /etc/network/interfaces.d/thunderbolt. If you set IP in frr I find that it fails when I restart networking (if you need to do that from time to time).

I reverted and improved upon @ronindesign solution. I liked the ability to get a fresh log in /tmp after boot (its deleted every boot, I do not need history). It seems that some few cases when the server is booting udev detects en05 before en06, and it runs ifup so quickly that en06 is not ready to "go up". This has happened once every 10-12 boots (I have been rigorously testing) so I needed to adapt. This is my current settings which works very well.

#/etc/udev/rules.d/10-tb-en.rules

ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-net.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-net.sh"

#/usr/local/bin/pve-en0[5-6].sh

#!/bin/bash

LOGFILE="/tmp/udev-debug.log"
VERBOSE="" # Set this to "-v" for verbose logging
IF="en05"
echo "$(date): pve-$IF.sh triggered by udev" >> "$LOGFILE"

# If multiple interfaces go up at the same time, 
# retry 10 times and break the retry when successful
for i in {1..10}; do
    echo "$(date): Attempt $i to bring up $IF" >> "$LOGFILE"
    
    /usr/sbin/ifup $VERBOSE $IF >> "$LOGFILE" 2>&1 && {
        echo "$(date): Successfully brought up $IF on attempt $i" >> "$LOGFILE"
        break
    }

    echo "$(date): Attempt $i failed, retrying in 3 seconds..." >> "$LOGFILE"
    sleep 3
done

Remember to change IF= to the desired interface, and also make 2 distinct files like described in the gist.

Example output in the log after a normal boot:

Fri Feb 14 13:47:01 CET 2025: pve-en06.sh triggered by udev
Fri Feb 14 13:47:01 CET 2025: pve-en05.sh triggered by udev
Fri Feb 14 13:47:01 CET 2025: Attempt 1 to bring up en06
Fri Feb 14 13:47:01 CET 2025: Attempt 1 to bring up en05
error: Another instance of this program is already running.
Fri Feb 14 13:47:01 CET 2025: Attempt 1 failed, retrying in 3 seconds...
Fri Feb 14 13:47:01 CET 2025: Successfully brought up en06 on attempt 1
Fri Feb 14 13:47:04 CET 2025: Attempt 2 to bring up en05
Fri Feb 14 13:47:04 CET 2025: Successfully brought up en05 on attempt 2

I reverted and improved upon @ronindesign solution. I liked the ability to get a fresh log in /tmp after boot (its deleted every boot, I do not need history). It seems that some few cases when the server is booting udev detects en05 before en06, and it runs ifup so quickly that en06 is not ready to "go up". This has happened once every 10-12 boots (I have been rigorously testing) so I needed to adapt. This is my current settings which works very well.

#/etc/udev/rules.d/10-tb-en.rules

ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-net.sh"
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en06", RUN+="/usr/local/bin/pve-net.sh"

#/usr/local/bin/pve-en0[5-6].sh

#!/bin/bash

LOGFILE="/tmp/udev-debug.log"
VERBOSE="" # Set this to "-v" for verbose logging
IF="en05"
echo "$(date): pve-$IF.sh triggered by udev" >> "$LOGFILE"

# If multiple interfaces go up at the same time, 
# retry 10 times and break the retry when successful
for i in {1..10}; do
    echo "$(date): Attempt $i to bring up $IF" >> "$LOGFILE"
    
    /usr/sbin/ifup $VERBOSE $IF >> "$LOGFILE" 2>&1 && {
        echo "$(date): Successfully brought up $IF on attempt $i" >> "$LOGFILE"
        break
    }

    echo "$(date): Attempt $i failed, retrying in 3 seconds..." >> "$LOGFILE"
    sleep 3
done

Remember to change IF= to the desired interface, and also make 2 distinct files like described in the gist.

Example output in the log after a normal boot:

Fri Feb 14 13:47:01 CET 2025: pve-en06.sh triggered by udev
Fri Feb 14 13:47:01 CET 2025: pve-en05.sh triggered by udev
Fri Feb 14 13:47:01 CET 2025: Attempt 1 to bring up en06
Fri Feb 14 13:47:01 CET 2025: Attempt 1 to bring up en05
error: Another instance of this program is already running.
Fri Feb 14 13:47:01 CET 2025: Attempt 1 failed, retrying in 3 seconds...
Fri Feb 14 13:47:01 CET 2025: Successfully brought up en06 on attempt 1
Fri Feb 14 13:47:04 CET 2025: Attempt 2 to bring up en05
Fri Feb 14 13:47:04 CET 2025: Successfully brought up en05 on attempt 2

I split the functions up with affinity. It’s really stable for me now. My screenshot on ceph page. Thank you for this.