dual-stack-openfabric-mesh.md

Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing

this gist is part of this series

This assumes you are running Proxmox 8.2 and that the line source /etc/network/interfaces.d/* is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).

This changes the previous file design thanks to @NRGNet for the suggestions to move thunderbolt settings to a file in /etc/network/interfaces.d it makes the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)

This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node

NOTES on Dual Stack

I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6. For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky....

Defining thunderbolt network

Create a new file using nano /etc/network/interfaces.d/thunderbolt and populate with the following Remember X should match you node number, so for example 1,2 or 3.

auto lo:0
iface lo:0 inet static
        address 10.0.0.8X/32
        
auto lo:6
iface lo:6 inet static
        address fc00::8X/128
        
allow-hotplug en05
iface en05 inet manual
        mtu 65520

allow-hotplug en06
iface en06 inet manual
        mtu 65520

Save file, repeat on each node.

Enable IPv4 and IPv6 forwarding

1. use nano /etc/sysctl.conf to open the file
2. uncomment #net.ipv6.conf.all.forwarding=1 (remove the # symbol)
3. uncomment #net.ipv4.ip_forward=1 (remove the # symbol)
4. save the file
5. issue reboot now for a complete reboot

FRR Setup

Install FRR

Install Free Range Routing (FRR) apt install frr

Enable the fabricd daemon

1. edit the frr daemons file (nano /etc/frr/daemons) to change fabricd=no to fabricd=yes
2. save the file
3. restart the service with systemctl restart frr

Mitigate FRR Timing Issues at Boot

Add post-up command to /etc/network/interfaces

sudo nano /etc/network/interfaces

Add post-up /usr/bin/systemctl restart frr.serviceas the last line in the file (this should go after the line that starts source)

NOTE for Minisforum MS-01 users

make the post-up line above read sleep 5 && post-up /usr/bin/systemctl restart frr.service instead this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.

Configure OpenFabric (perforn on all nodes)

1. enter the FRR shell with vtysh
2. optionally show the current config with show running-config
3. enter the configure mode with configure
4. Apply the bellow configuration (it is possible to cut and paste this into the shell instead of typing it manually, you may need to press return to set the last !. Also check there were no errors in repsonse to the paste text.).

Note: the X should be the number of the node you are working on, as an example - 0.0.0.1, 0.0.0.2 or 0.0.0.3.

ip forwarding
ipv6 forwarding
!
interface en05
ip router openfabric 1
ipv6 router openfabric 1
exit
!
interface en06
ip router openfabric 1
ipv6 router openfabric 1
exit
!
interface lo
ip router openfabric 1
ipv6 router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0000.0000.000X.00
exit
!

5. you may need to pres return after the last ! to get to a new line - if so do this

6. exit the configure mode with the command end

7. save the configu with write memory

8. show the configure applied correctly with show running-config - note the order of the items will be different to how you entered them and thats ok. (If you made a mistake i found the easiest way was to edt /etc/frr/frr.conf - but be careful if you do that.)

9. use the command exit to leave setup

10. repeat steps 1 to 9 on the other 3 nodes

11. once you have configured all 3 nodes issue the command vtysh -c "show openfabric topology" if you did everything right you will see:

Area 1:
IS-IS paths to level-2 routers that speak IP
Vertex               Type         Metric Next-Hop             Interface Parent
pve1                                                                  
10.0.0.81/32         IP internal  0                                     pve1(4)
pve2                 TE-IS        10     pve2                 en06      pve1(4)
pve3                 TE-IS        10     pve3                 en05      pve1(4)
10.0.0.82/32         IP TE        20     pve2                 en06      pve2(4)
10.0.0.83/32         IP TE        20     pve3                 en05      pve3(4)

IS-IS paths to level-2 routers that speak IPv6
Vertex               Type         Metric Next-Hop             Interface Parent
pve1                                                                  
fc00::81/128         IP6 internal 0                                     pve1(4)
pve2                 TE-IS        10     pve2                 en06      pve1(4)
pve3                 TE-IS        10     pve3                 en05      pve1(4)
fc00::82/128         IP6 internal 20     pve2                 en06      pve2(4)
fc00::83/128         IP6 internal 20     pve3                 en05      pve3(4)

IS-IS paths to level-2 routers with hop-by-hop metric
Vertex               Type         Metric Next-Hop             Interface Parent

Now you should be in a place to ping each node from evey node across the thunderbolt mesh using IPv4 or IPv6 as you see fit.

My Routing look like this.

When i try to setup ceph i cannot add the monitors on ipv4 or ipv6

Just wondering if trying to add the thunderbolt networking to an existing cluster would cause issues? I have MS-01's with a spare NVME in each that I could use for CEPH, but I'm unsure if this approach would require me to start from scratch then restore some PBS backups?

Also is anyone using Kernel 6.8.12-4-pve?

Thanks for the warning, great catch! I did some more tests and changed the BindsTo to Wants in the dependencies.conf this solved the case that frr is shutdown on other nodes if one goes down.

[Unit]
Wants=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device
After=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device

Thanks for this. It seems it solved some startup issues I was having on my MS-01 nodes. I'll have to do some more tests to be sure.

Just wondering if trying to add the thunderbolt networking to an existing cluster would cause issues? I have MS-01's with a spare NVME in each that I could use for CEPH, but I'm unsure if this approach would require me to start from scratch then restore some PBS backups?

Also is anyone using Kernel 6.8.12-4-pve?

Thanks for the warning, great catch! I did some more tests and changed the BindsTo to Wants in the dependencies.conf this solved the case that frr is shutdown on other nodes if one goes down.

[Unit]
Wants=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device
After=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device

Thanks for this. It seems it solved some startup issues I was having on my MS-01 nodes. I'll have to do some more tests to be sure.

Coming back to confirm following this GIST with an existing cluster worked fine. No issues so far and I can ping IPV6 back and fourth.

I used the systemd method quoted instead of the wait method in network services. I'll report back if there's any issues with CEPH.

So...I cant even get my thunderbot-net adapters to show up when I do ip a lol

lspci shows the 2 Raptor Lake-P Thunderbolt 4 ports on the 2 units I have. I used a Dell Thunderbolt Dock that I have to validate the PCI path and recorded the below for both MS-01s.

TB port 1: /devices/pci0000:00/0000:00:0d.2

TB4 port 2: /devices/pci0000:00/0000:00:0d.3

I did see the Thunderbolt0 and Thunderbolt1 in the nic table for a minute but then I went thru the 2 steps around the scripts and IP forwarding that required reboots, I think thats where they were lost.

I was able to get ALL the way to the end steps with FRR and this is the output:

root@pve1:~# vtysh -c "show openfabric topology"
Area 1:
IS-IS paths to level-2 routers that speak IP
Vertex Type Metric Next-Hop Interface Parent
pve1
10.0.0.81/32 IP internal 0 pve1(4)

IS-IS paths to level-2 routers that speak IPv6
Vertex Type Metric Next-Hop Interface Parent
pve1
fc00::81/128 IP6 internal 0 pve1(4)

IS-IS paths to level-2 routers with hop-by-hop metric
Vertex Type Metric Next-Hop Interface Parent

---

root@pve2:/etc/systemd/system# vtysh -c "show openfabric topology"
Area 1:
IS-IS paths to level-2 routers that speak IP
Vertex Type Metric Next-Hop Interface Parent
pve2
10.0.0.82/32 IP internal 0 pve2(4)

IS-IS paths to level-2 routers that speak IPv6
Vertex Type Metric Next-Hop Interface Parent
pve2
fc00::82/128 IP6 internal 0 pve2(4)

IS-IS paths to level-2 routers with hop-by-hop metric
Vertex Type Metric Next-Hop Interface Parent

Any thoughts as to what the problem might be? I know its kind of an open ended question

The only other thing I found odd is that I was following this YT Video: https://www.youtube.com/watch?v=Tb3KH2RbsTE&t=202s
In the previous gist steps, it said put the auto lo: configuration into the interfaces.d/thunderbolt but I've never been able to get anything to show up in there. Things only show up when I put it in the primary interfaces configuration file, like YT video states.

Thoughts?

So...I cant even get my thunderbot-net adapters to show up when I do ip a lol

lspci shows the 2 Raptor Lake-P Thunderbolt 4 ports on the 2 units I have. I used a Dell Thunderbolt Dock that I have to validate the PCI path and recorded the below for both MS-01s.

TB port 1: /devices/pci0000:00/0000:00:0d.2

TB4 port 2: /devices/pci0000:00/0000:00:0d.3

I did see the Thunderbolt0 and Thunderbolt1 in the nic table for a minute but then I went thru the 2 steps around the scripts and IP forwarding that required reboots, I think thats where they were lost.

I was able to get ALL the way to the end steps with FRR and this is the output:

root@pve1:~# vtysh -c "show openfabric topology" Area 1: IS-IS paths to level-2 routers that speak IP Vertex Type Metric Next-Hop Interface Parent pve1 10.0.0.81/32 IP internal 0 pve1(4)

IS-IS paths to level-2 routers that speak IPv6 Vertex Type Metric Next-Hop Interface Parent pve1 fc00::81/128 IP6 internal 0 pve1(4)

IS-IS paths to level-2 routers with hop-by-hop metric Vertex Type Metric Next-Hop Interface Parent

root@pve2:/etc/systemd/system# vtysh -c "show openfabric topology" Area 1: IS-IS paths to level-2 routers that speak IP Vertex Type Metric Next-Hop Interface Parent pve2 10.0.0.82/32 IP internal 0 pve2(4)

IS-IS paths to level-2 routers that speak IPv6 Vertex Type Metric Next-Hop Interface Parent pve2 fc00::82/128 IP6 internal 0 pve2(4)

IS-IS paths to level-2 routers with hop-by-hop metric Vertex Type Metric Next-Hop Interface Parent

Any thoughts as to what the problem might be? I know its kind of an open ended question

The only other thing I found odd is that I was following this YT Video: https://www.youtube.com/watch?v=Tb3KH2RbsTE&t=202s In the previous gist steps, it said put the auto lo: configuration into the interfaces.d/thunderbolt but I've never been able to get anything to show up in there. Things only show up when I put it in the primary interfaces configuration file, like YT video states.

Thoughts?

For me it works fine until I reboot one of the nodes and then I see the same thing you are seeing (ip a doesn't show en05 on the list and each of the nodes will only show their own details with vtysh -c "show openfabric topology"). When that happens nothing brings it back. I just have to reboot multiple times and sometimes I get the network back. I'ved tried every script and delay config (individually not all at the same time) with the same result. I've also tried 3 different brand TB cables (Anker, Cable Matters and the last one I got is the OWC one) and can't get more than 16Gbits/sec average (with tons of retries - in the hundreds per interval) but I only have TB 3 ports on my NUC11 and I read the TB fixes are only applicable to TB4. I have IOMMU enabled and working so that's definitely not the issue but I am on the latest kernel too so that and the fact I'm using TB3 might not allow me to get faster speeds. However, 16 is fine with me since I'm moving from a 1G connection so it's still faster than my normal ETH network migrates but it's just not stable in my case so I moved my migration network back to ETH (for now)

I never got the /etc/network/interfaces.d/thunderbolt config to work so I just added all the config options into /etc/network/interfaces which works fine.

First of all, I'd like to thank you for the comprehensive guide; this is extremely helpful!

Let me share my experience. I have finally gotten it working, and here is what worked for me.

Hardware: 2-node cluster with Intel NUC 12 i7 and Asus NUC 13 i5 + Corosync QDevice on Raspberry Pi 4
Software: Proxmox 8.3.2 + Ceph Quincy

Everything works perfectly according to the instructions (thanks again!), though there are a few points worth mentioning:

• The only way to make the mesh rise after reboot is via /etc/systemd/system/frr.service.d/dependencies.conf (as mentioned in the comments above).

• The first attempt yielded about 10 Gbit/s and several thousand retries. After setting the IOMMU option in GRUB, I achieved 13-17 Gbit/s. Affinity to performance CPU cores alone also gave 14-17 Gbit/s. However, combining both the IOMMU option and CPU affinity finally resulted in a stable 26 Gbit/s with minimal retries.

• One more thing - there is a bug in Ceph Quincy. If the public monitor interface is on IPv6, the health checker shows the error 'public ceph network is not available'. You can safely ignore this error. (Reference: https://tracker.ceph.com/issues/67517)

Hardware: 2-node cluster with Intel NUC 12 i7 and Asus NUC 13 i5 + Corosync QDevice on Raspberry Pi 4
Software: Proxmox 8.3.2 + Ceph Quincy

So it's possible to have a 2-node cluster (with QDevice) work with Ceph? I read in every guide that Ceph required at least 3 nodes.

I'm using 2 MS-01 (i9-13900H) with Proxmox replication, because 3 nodes is way overkill for my homelab. The nodes are idling most of the time so I don't want to add a 3rd node just to have Ceph.

Hardware: 2-node cluster with Intel NUC 12 i7 and Asus NUC 13 i5 + Corosync QDevice on Raspberry Pi 4
Software: Proxmox 8.3.2 + Ceph Quincy

So it's possible to have a 2-node cluster (with QDevice) work with Ceph? I read in every guide that Ceph required at least 3 nodes.

I'm using 2 MS-01 (i9-13900H) with Proxmox replication, because 3 nodes is way overkill for my homelab. The nodes are idling most of the time so I don't want to add a 3rd node just to have Ceph.

Yeah, Ceph requires three nodes to work correctly.

For those of you that have this cluster working correctly, have you updated Proxmox to the latest version and if so, has there been any issues with it breaking the thunderbolt network?

Hardware: 2-node cluster with Intel NUC 12 i7 and Asus NUC 13 i5 + Corosync QDevice on Raspberry Pi 4
Software: Proxmox 8.3.2 + Ceph Quincy

So it's possible to have a 2-node cluster (with QDevice) work with Ceph? I read in every guide that Ceph required at least 3 nodes.
I'm using 2 MS-01 (i9-13900H) with Proxmox replication, because 3 nodes is way overkill for my homelab. The nodes are idling most of the time so I don't want to add a 3rd node just to have Ceph.

Yeah, Ceph requires three nodes to work correctly.

I have Ceph working with 2 MS-01 nodes and a QDevice configured with 1 replica (basically raid 1) and its working fine. 4 OSDs, 97 PGs, 25Gbit/s replication. Currently running on Proxmox 8.3.2 Ceph Squid

So it's possible to have a 2-node cluster (with QDevice) work with Ceph? I read in every guide that Ceph required at least 3 nodes.

Absolutely. Although this is highly unrecommended, it certainly works. The trick is that the UI doesn't allow you to create a monitor with a minimum of 1 node for read and write operations, but it's possible to change it later, after creation. You can create a monitor with default settings and immediately afterward adjust it to two nodes normal and 1 node minimum. It shows a warning but works.

Moreover, it's even possible to create a Ceph cluster with only one disk and add the second disk on the second node later. I did this because I had some data on one node that I wanted to move to the Ceph pool. So I created a pool with one disk first, then moved the data to the pool from the disk on the second node, and added the second disk afterward.

For those of you that have this cluster working correctly, have you updated Proxmox to the latest version and if so, has there been any issues with it breaking the thunderbolt network?

8.3.2 works fine with both iommu and CPU affinity configured.

The trick is that the UI doesn't allow you to create a monitor with a minimum of 1 node for read and write operations, but it's possible to change it later, after creation.

but if it doesn't allow you to create it, how do you create it first, in order to change it later?

So I created a pool with one disk first, then moved the data to the pool from the disk on the second node, and added the second disk afterward.

good to know, because I will have the same issue. How do you move data from the other node when you have the ceph volume only on the other?

I had some frr boot timing issues with my Minisforum MS-01 as described in the gist. However adding the following line to the /etc/network/interfaces was not working reliably for me. sleep 5 && post-up /usr/bin/systemctl restart frr.service Sometimes according to dmesg it took quite some time until the interfaces en05 and en06 where correctly set up. This for some reason lead to only ipv6 routing being initialized but not ipv4 (not sure why here).
So I added the following file /etc/systemd/system/frr.service.d/dependencies.conf with this content:

[Unit]
BindsTo=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device
After=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device

This ensures that frr only starts after en05 and en06 are fully initialized. This worked great for me and the post-up command is not needed anymore in the interfaces file. I thought I would share this if someone else had similar problems. I would also be happy to hear your thoughts on this solution.

@nimro27 Thank you so much! This solved it for me on my 3 node MS-01 cluster!

When I add this file the frr.service does not even start anymore, not on boot and not manually. Any idea?

@FrankSchoene @nimro27 Just a warning, when I attempted to use this approach (dependencies.conf) is caused a couple problems. It would occasionally prevent en05/06 from coming up during boot (logging dependency errors), but more importantly it would also cause the frr service on all other nodes to shutdown when the current node was rebooting. In my case, I had to remove the dependencies.conf and go back to using a post-up with sleep 10. One minor difference in my case, I used a script in /etc/network/if-up.d/ instead of explicitly adding post-up to the interfaces file. Everything seems to be working well now, and survives reboots of any node. I'm running 3 MS-01's.

Thanks for the warning, great catch! I did some more tests and changed the BindsTo to Wants in the dependencies.conf this solved the case that frr is shutdown on other nodes if one goes down.

[Unit]
Wants=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device
After=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device

However I could not replicate the case where en05/06 would not come up during boot. Could you explain the logging dependency error a bit more? Is it related to the shutdown issue with BindsTo?

I had some frr boot timing issues with my Minisforum MS-01 as described in the gist. However adding the following line to the /etc/network/interfaces was not working reliably for me. sleep 5 && post-up /usr/bin/systemctl restart frr.service Sometimes according to dmesg it took quite some time until the interfaces en05 and en06 where correctly set up. This for some reason lead to only ipv6 routing being initialized but not ipv4 (not sure why here).
So I added the following file /etc/systemd/system/frr.service.d/dependencies.conf with this content:

[Unit]
BindsTo=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device
After=sys-subsystem-net-devices-en05.device sys-subsystem-net-devices-en06.device

This ensures that frr only starts after en05 and en06 are fully initialized. This worked great for me and the post-up command is not needed anymore in the interfaces file. I thought I would share this if someone else had similar problems. I would also be happy to hear your thoughts on this solution.

@nimro27 Thank you so much! This solved it for me on my 3 node MS-01 cluster!

When I add this file the frr.service does not even start anymore, not on boot and not manually. Any idea?

@FrankSchoene @nimro27 Just a warning, when I attempted to use this approach (dependencies.conf) is caused a couple problems. It would occasionally prevent en05/06 from coming up during boot (logging dependency errors), but more importantly it would also cause the frr service on all other nodes to shutdown when the current node was rebooting. In my case, I had to remove the dependencies.conf and go back to using a post-up with sleep 10. One minor difference in my case, I used a script in /etc/network/if-up.d/ instead of explicitly adding post-up to the interfaces file. Everything seems to be working well now, and survives reboots of any node. I'm running 3 MS-01's.

Another approach I found that worked to troubleshoot issues with FRRouting not starting at boot was to change the frr.service config for FRR service to wait until the network is fully loaded.

Issue: I could find my devices when running vtysh -c "show openfabric topology" and checked my network logs and found "Node2 networking[1170509]: error: /etc/network/interfaces: line31: error processing line 'post-up sleep 5 && /usr/bin/systemctl restart frr.service'" in my logs

Solutions: Change the frr.service config for FRR service to wait until the network is fully loaded using "network-online.target"

How:
nano /lib/systemd/system/frr.service and update to use network-online.target and change Wants= and After= instead of network.target it would look something like this after changing :

[Unit]
Description=FRRouting
Documentation=https://frrouting.readthedocs.io/en/latest/setup.html
Wants=network-online.target
After=network-online.target systemd-sysctl.service
OnFailure=heartbeat-failed@%n

Save and exit the file. remove or commet out sleep 5 && post-up /usr/bin/systemctl restart frr.service
and then restart network services "systemctl restart networking" and then run "journalctl -xe | grep networking" to see if it worked. I have done this on 2 of my nodes and the MS-01 is running the BindsTo solution above. I hope this helps someone, been at this for 3 days and finally got my mesh to work - 2x thunderbolt and 1 on SFP+

I managed to set up the 3 machines with Thunderbolt as the migration network - I think managed to get about 8Gb / sec transfer when moving a VM between them. Not bad for machines that are over 10 years old! This in itself is VERY helpful!!

I did not get the ip forwarding to appear as visible in the vtysh 'runining config'

I tried unplugging my main ethernet cable to one of the servers and I was hoping the internet connection would automatically switch over to using the Thunderbolt connection - but this did not happen unfortunately.

Hi @ricksteruk,

i am trying to get this going between two Mac Minis A1347, therefore also Thunderbolt 2. Since you seemingly made it happen on macs as well, you might have a clue whats going on here:

Both machines do see each other via LLDP as neighbour but only if i force LLDPD to emit LLDP-MED frames, setting the class option -M 4 aka Network Connectivity Device. This is also reliable, if i disconnect the cable on either machine and reconnect, the LLDP comes up and along with it the en05 network devices, with status 'up' (configured as per @scyto 's guide).

But as soon as i start the frr service, LLDP looses connection. This also freezes the frr daemon on one machine till i physically interrupt the cable connection by pulling cable or by issuing ifdown en05 on the opposite machine on which frr daemon is not frozen.

For reference i added the output of get-thunderbolt-config.sh of both machines below.

Host pve1

=========================================
Thunderbolt Mesh Network Config Info Tool
=========================================

-----------------------------------------
Kernel Version
-----------------------------------------
Linux pve 6.8.12-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-4 (2024-11-06T15:04Z) x86_64 GNU/Linux

-----------------------------------------
File: /etc/network/interfaces
-----------------------------------------
auto lo
iface en05 inet manual
iface lo inet loopback
iface enp2s0f0 inet manual
auto vmbr0
iface vmbr0 inet static
	address 10.1.1.121/24
	gateway 10.1.1.1
	bridge-ports enp2s0f0
	bridge-stp off
	bridge-fd 0
post-up /usr/bin/systemctl restart frr.service
source /etc/network/interfaces.d/*

-----------------------------------------
File: /etc/network/interfaces.d/thunderbolt
-----------------------------------------
auto lo
iface lo inet static
        address 10.0.0.81/32

auto lo:6
iface lo:6 inet static
        address fc00::81/128

allow-hotplug en05
iface en05 inet manual
        mtu 65520

----------------------------------------
File: /usr/local/bin/pve-en05.sh
-rwxr-xr-x 1 root root 154 Jan  9 22:51 /usr/local/bin/pve-en05.sh
-----------------------------------------
/usr/sbin/ifup en05

----------------------------------------
File: /usr/local/bin/pve-en06.sh
ls: cannot access '/usr/local/bin/pve-en06.sh': No such file or directory
-----------------------------------------
grep: /usr/local/bin/pve-en06.sh: No such file or directory

-----------------------------------------
File: /etc/modules
-----------------------------------------
thunderbolt

-----------------------------------------
File: /etc/systemd/network/00-thunderbolt0.link
-----------------------------------------
[Match]
Path=pci-0000:08:00.0
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05

-----------------------------------------
File: /etc/systemd/network/00-thunderbolt1.link
-----------------------------------------
grep: /etc/systemd/network/00-thunderbolt1.link: No such file or directory

-----------------------------------------
File: /etc/sysctl.conf
-----------------------------------------
et.ipv4.ip_forward=1
et.ipv6.conf.all.forwarding=1

-----------------------------------------
File: /etc/udev/rules.d/10-tb-en.rules
-----------------------------------------
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"

-----------------------------------------
File: /etc/frr/frr.conf
-----------------------------------------
frr version 10.2.1
frr defaults traditional
hostname pve
log syslog errors
service integrated-vtysh-config
!
interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
exit
!
interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive
exit
!
router openfabric 1
 net 49.0000.0000.0001.00
 fabric-tier 0
exit
!

-----------------------------------------
File: /etc/frr/daemons
-----------------------------------------
bgpd=no
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
pim6d=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no
fabricd=yes
vrrpd=no
pathd=no
vtysh_enable=yes
zebra_options="  -A 127.0.0.1 -s 90000000"
mgmtd_options="  -A 127.0.0.1"
bgpd_options="   -A 127.0.0.1"
ospfd_options="  -A 127.0.0.1"
ospf6d_options=" -A ::1"
ripd_options="   -A 127.0.0.1"
ripngd_options=" -A ::1"
isisd_options="  -A 127.0.0.1"
pimd_options="   -A 127.0.0.1"
pim6d_options="  -A ::1"
ldpd_options="   -A 127.0.0.1"
nhrpd_options="  -A 127.0.0.1"
eigrpd_options=" -A 127.0.0.1"
babeld_options=" -A 127.0.0.1"
sharpd_options=" -A 127.0.0.1"
pbrd_options="   -A 127.0.0.1"
staticd_options="-A 127.0.0.1"
bfdd_options="   -A 127.0.0.1"
fabricd_options="-A 127.0.0.1"
vrrpd_options="  -A 127.0.0.1"
pathd_options="  -A 127.0.0.1"

-----------------------------------------
Command: vtysh -c "show openfabric topology"
-----------------------------------------
Area 1:
IS-IS paths to level-2 routers that speak IP
 Vertex        Type         Metric  Next-Hop  Interface  Parent
 ----------------------------------------------------------------
 pve
 10.0.0.81/32  IP internal  0                            pve(4)


IS-IS paths to level-2 routers that speak IPv6
 Vertex        Type          Metric  Next-Hop  Interface  Parent
 -----------------------------------------------------------------
 pve
 fc00::81/128  IP6 internal  0                            pve(4)


IS-IS paths to level-2 routers with hop-by-hop metric
 Vertex  Type  Metric  Next-Hop  Interface  Parent




-----------------------------------------
Command: vtysh -c "show running-config"
-----------------------------------------
Building configuration...

Current configuration:
!
frr version 10.2.1
frr defaults traditional
hostname pve
log syslog errors
no ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
exit
!
interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive
exit
!
router openfabric 1
 net 49.0000.0000.0001.00
 fabric-tier 0
exit
!
end

----------------------------------------
File: /etc/network/if-up.d/thunderbolt-affinity
ls: cannot access '/etc/network/if-up.d/thunderbolt-affinity': No such file or directory
-----------------------------------------

Host pve2

=========================================
Thunderbolt Mesh Network Config Info Tool
=========================================

-----------------------------------------
Kernel Version
-----------------------------------------
Linux pve2 6.8.12-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-4 (2024-11-06T15:04Z) x86_64 GNU/Linux

-----------------------------------------
File: /etc/network/interfaces
-----------------------------------------
auto lo
iface en05 inet manual
iface lo inet loopback
iface enp1s0f0 inet manual
auto vmbr0
iface vmbr0 inet static
	address 10.1.1.122/24
	gateway 10.1.1.1
	bridge-ports enp1s0f0
	bridge-stp off
	bridge-fd 0
post-up /usr/bin/systemctl restart frr.service
source /etc/network/interfaces.d/*

-----------------------------------------
File: /etc/network/interfaces.d/thunderbolt
-----------------------------------------
auto lo
iface lo inet static
        address 10.0.0.82/32

auto lo:6
iface lo:6 inet static
        address fc00::82/128

allow-hotplug en05
iface en05 inet manual
        mtu 65520

----------------------------------------
File: /usr/local/bin/pve-en05.sh
-rwxr-xr-x 1 root root 154 Jan  9 22:51 /usr/local/bin/pve-en05.sh
-----------------------------------------
/usr/sbin/ifup en05

----------------------------------------
File: /usr/local/bin/pve-en06.sh
ls: cannot access '/usr/local/bin/pve-en06.sh': No such file or directory
-----------------------------------------
grep: /usr/local/bin/pve-en06.sh: No such file or directory

-----------------------------------------
File: /etc/modules
-----------------------------------------
thunderbolt

-----------------------------------------
File: /etc/systemd/network/00-thunderbolt0.link
-----------------------------------------
[Match]
Path=pci-0000:06:00.0
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05

-----------------------------------------
File: /etc/systemd/network/00-thunderbolt1.link
-----------------------------------------
grep: /etc/systemd/network/00-thunderbolt1.link: No such file or directory

-----------------------------------------
File: /etc/sysctl.conf
-----------------------------------------
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

-----------------------------------------
File: /etc/udev/rules.d/10-tb-en.rules
-----------------------------------------
ACTION=="move", SUBSYSTEM=="net", KERNEL=="en05", RUN+="/usr/local/bin/pve-en05.sh"

-----------------------------------------
File: /etc/frr/frr.conf
-----------------------------------------
frr version 10.2.1
frr defaults traditional
hostname pve2
log syslog errors
service integrated-vtysh-config
!
interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
exit
!
interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive
exit
!
router openfabric 1
 net 49.0000.0000.0002.00
 fabric-tier 0
exit
!

-----------------------------------------
File: /etc/frr/daemons
-----------------------------------------
bgpd=no
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
pim6d=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no
fabricd=yes
vrrpd=no
pathd=no
vtysh_enable=yes
zebra_options="  -A 127.0.0.1 -s 90000000"
mgmtd_options="  -A 127.0.0.1"
bgpd_options="   -A 127.0.0.1"
ospfd_options="  -A 127.0.0.1"
ospf6d_options=" -A ::1"
ripd_options="   -A 127.0.0.1"
ripngd_options=" -A ::1"
isisd_options="  -A 127.0.0.1"
pimd_options="   -A 127.0.0.1"
pim6d_options="  -A ::1"
ldpd_options="   -A 127.0.0.1"
nhrpd_options="  -A 127.0.0.1"
eigrpd_options=" -A 127.0.0.1"
babeld_options=" -A 127.0.0.1"
sharpd_options=" -A 127.0.0.1"
pbrd_options="   -A 127.0.0.1"
staticd_options="-A 127.0.0.1"
bfdd_options="   -A 127.0.0.1"
fabricd_options="-A 127.0.0.1"
vrrpd_options="  -A 127.0.0.1"
pathd_options="  -A 127.0.0.1"

-----------------------------------------
Command: vtysh -c "show openfabric topology"
-----------------------------------------
Area 1:
IS-IS paths to level-2 routers that speak IP
 Vertex        Type         Metric  Next-Hop  Interface  Parent
 -----------------------------------------------------------------
 pve2
 10.0.0.82/32  IP internal  0                            pve2(4)


IS-IS paths to level-2 routers that speak IPv6
 Vertex        Type          Metric  Next-Hop  Interface  Parent
 ------------------------------------------------------------------
 pve2
 fc00::82/128  IP6 internal  0                            pve2(4)


IS-IS paths to level-2 routers with hop-by-hop metric
 Vertex  Type  Metric  Next-Hop  Interface  Parent




-----------------------------------------
Command: vtysh -c "show running-config"
-----------------------------------------
Building configuration...

Current configuration:
!
frr version 10.2.1
frr defaults traditional
hostname pve2
log syslog errors
service integrated-vtysh-config
!
interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
exit
!
interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive
exit
!
router openfabric 1
 net 49.0000.0000.0002.00
 fabric-tier 0
exit
!
end

----------------------------------------
File: /etc/network/if-up.d/thunderbolt-affinity
ls: cannot access '/etc/network/if-up.d/thunderbolt-affinity': No such file or directory
-----------------------------------------

Deleted - as this was not correct - see my updated and working config here

Hi @scyto

Thank you very much for your work and to everyone here contributing in the comments. This helped me a lot.

I want to help out and give back, so I want to point out a few things for those trying this on MS-01 and maybe help others too.

You have a mistake in the post-up line for MS-01. It is not sleep 5 && post-up ..... as post-up is the marker to trigger the command while loading settings from the /etc/network/interfaces file. To make it work it should be:
post-up sleep 5 && /usr/bin/systemctl restart frr.service

Otherwise it errors out as it does not know what "sleep" is . That's why people are reporting it is not working on MS01.

IPv6 is working perfectly, but I gave up on IPv4, as frr does not start, might work with the solutions in the comment, but I might not have implemented them correctly. Who knows.

Also, I just added the thunderbolt interfaces in the /etc/network/interfaces file directly, as for some reason, my install on Proxmox 8.3 does not load the sources ..... at least it does not seam too. Too tired to see why.
Last but not least, a huge thank you to @Allistah for the thunderbolt-affinity solution, it worked wonders. For MS-01, with i9 13900H, as there are 6 cores and 12 threads, it shoudl have 0-11 in it, but works very well with 0-5 using half of the P core threads.

I think this was my first post on github, and it is 3 in the morning, so please excuse the rudimentary formating. Just wanted to help out so I don't forget in the morning.

Cheers,
Vlad

Hi @scyto

Based on my testing in my environment, it appears that specifying a sequence of shell commands with post-up is not possible.
It would be better to place a shell script in /etc/network/if-up.d/ and have it executed when en05 and en06 are brought up.

nano /etc/network/if-up.d/frr

#!/bin/bash

if [[ $IFACE == "en05" ]] || [[ $IFACE == "en06" ]]; then
 sleep 5 && /usr/bin/systemctl restart frr.service
fi

chmod +x /etc/network/if-up.d/frr

With this method, IPv4 can now be used stably.

---

Also, if anyone set FRR rules via the command line, when using Proxmox VE SDN, applying the SDN seems to overwrite the Thunderbolt Networking rules set in this article, effectively removing them.

If you write the FRR configuration for Thunderbolt Networking into the following file, it will merge with the SDN apply.

nano /etc/frr/frr.conf.local

Does anyone have a consistently functional FRR config using 3 MS-01 boxes? I have implemented several of the changes in the comments here and am still having issues getting en05 and en06 to come up after a reboot.