setup openfabric for thunderbolt / ceph network

openfabric-mesh-net.md

Enable IPv6 and IPv4

NOTE: as of 9/14/2023 this will only work with a custom patched linux kernel6.5.2 Proxmox have indicated they will back port these patches contact me on the proxmox forum if you wish to roll the patches yourself

Key Parameters

Nodel:

• IPv4 Address = 10.0.0.81/32
• IPv6 Address = fc00::81/128
• net = net 49.0001.1111.1111.1111.00

Node2:

• IPv4 Address = 10.0.0.82/32
• IPv6 Address = fc00:82/128
• net = net 49.0001.2222.2222.2222.00

Node3:

• IPv4 Address = 10.0.0.83/32
• IPv6 Address = fc00::83/128
• net = net 49.0001.3333.3333.3333.00

Enable IP forwarding

Enable for both IPv4 and IPv6 - note ceph should be used on either IPv4 ot IPv6. Dual stack is not supported and in my testing was a little funky... i will cover this later...

1. use nano /etc/sysctl.conf to open the file
2. uncomment #net.ipv6.conf.all.forwarding=1 (remove the # symbol)
3. uncomment #3net.ipv4.ip_forward=1 (remove the # symbol)
4. save the file

Install Free Range Routing (FRR)

1. Install Free Range Routing (FRR) with apt install frr
2. Edit the FRR config file: nano /etc/frr/daemons
3. Adjust fabricd=no to fabricd=yes
4. save the file
5. restart the service with systemctl restart frr

Configure fabricd (perforn on all nodes)

Configure FRR

1. enter the FRR shell with vtysh
2. optionally show the current config with show running-config
3. enter the configure mode with configure
4. Apply the bellow configuration (it is possible to cut and paste this into the shell instead of typing it manually, you may need to press return to set the last !. Also check there were no errors in repsonse to the paste text.). Note: use the key data from the earlier section

ip forwarding
#ipv6 forwarding
!
interface lo
 ip address 10.0.0.81/32
 ip router openfabric 1
 ipv6 address 2600:a801:830:e1::81/128
 ipv6 router openfabric 1
 openfabric passive
exit
!
interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
exit
!
interface en06
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric csnp-interval 2
 openfabric hello-interval 1
 openfabric hello-multiplier 2
exit
!
router openfabric 1
 net 49.0001.1111.1111.1111.00
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
exit
!

5. you may need to pres return after the last ! to get to a new line - if so do this
6. exit the configure mode with the command end
7. save the configu with write memory
8. show the configure applied correctly with show running-config - note the order of the items will be different to how you entered them and thats ok. (If you made a mistake i found the easiest way was to edt /etc/frr/frr.conf - but be careful if you do that as this can sometimes corrupt the file - even if there is nothing visbly wrong, and do not create the file by copying and pasting into it.)
9. use the command exit to leave setup
10. repeat steps 1 to 10 on the other 2 nodes

Once all nodes have been cofnigured perform the following on each noe

1. ifreload -a
2. systemctl restart frr

then on any node sheck the open frabric as follows

1. vtysh
2. show openfabric route

and you should see something like:

pve2# show openfabric route 
Area 1:
IS-IS L2 IPv4 routing table:

 Prefix        Metric  Interface  Nexthop    Label(s)  
 ------------------------------------------------------
 10.0.0.81/32  20      en05       10.0.0.81  -         
 10.0.0.82/32  0       -          -          -         
 10.0.0.83/32  20      en06       10.0.0.83  -      

note at this time after any boot even i have found one needs to reissue the ifreload and systemctl commands above on each node - i haven't found an elegant way to solve this (if you add the restart command to pve-05.sh and pve-06.sh you risk invalidating the service as it will have had too many restarts too fast).