THIS GIST IS NOW DEPRECATED NEW ONE IS AVAILABLE HERE I WONT BE UPDATING THIS ONE OR REPLYING TO COMMENTS ON THIS ONE (COMMENTS NOW DISABLED).
this gist is part of this series
This assumes you are running Proxmox 8.2 and that the line source /etc/network/interfaces.d/* is at the end of the interfaces file (this is automatically added to both new and upgraded installations of Proxmox 8.2).
This changes the previous file design thanks to @NRGNet for the suggestions to move thunderbolt settings to a file in /etc/network/interfaces.d it makes the system much more reliable in general, more maintainable esp for folks using IPv4 on the private cluster network (i still recommend the use of the IPv6 FC00 network you will see in these docs)
This will result in an IPv4 and IPv6 routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node
I have included this for completeness, i only run the FC00:: IPv6 network as ceph does not support dual stack, i strongly recommend you consider only using IPv6. For example for ceph do not dual stack - either use IPv4 or IPv6 addressees for all the monitors, MDS and daemons - despite the docs implying it is ok my findings on quincy are is it is funky....
Create a new file using nano /etc/network/interfaces.d/thunderbolt and populate with the following
Remember X should match you node number, so for example 1,2 or 3.
auto lo:0
iface lo:0 inet static
address 10.0.0.8X/32
auto lo:6
iface lo:6 inet static
address fc00::8X/128
allow-hotplug en05
iface en05 inet manual
mtu 65520
allow-hotplug en06
iface en06 inet manual
mtu 65520
Save file, repeat on each node.
- use
nano /etc/sysctl.confto open the file - uncomment
#net.ipv6.conf.all.forwarding=1(remove the # symbol) - uncomment
#net.ipv4.ip_forward=1(remove the # symbol) - save the file
- issue
reboot nowfor a complete reboot
Install Free Range Routing (FRR) apt install frr
- edit the frr daemons file (
nano /etc/frr/daemons) to changefabricd=notofabricd=yes - save the file
- restart the service with
systemctl restart frr
sudo nano /etc/network/interfaces
Add post-up /usr/bin/systemctl restart frr.serviceas the last line in the file (this should go after the line that starts source)
make the post-up line above read post-up sleep 5 && /usr/bin/systemctl restart frr.service instead
this has been verified to be required due to timing issues see on those units, exact cause unknown, may be needed on other hardware too.
- enter the FRR shell with
vtysh - optionally show the current config with
show running-config - enter the configure mode with
configure - Apply the bellow configuration (it is possible to cut and paste this into the shell instead of typing it manually, you may need to press return to set the last !. Also check there were no errors in repsonse to the paste text.).
Note: the X should be the number of the node you are working on, as an example - 0.0.0.1, 0.0.0.2 or 0.0.0.3.
ip forwarding
ipv6 forwarding
!
interface en05
ip router openfabric 1
ipv6 router openfabric 1
exit
!
interface en06
ip router openfabric 1
ipv6 router openfabric 1
exit
!
interface lo
ip router openfabric 1
ipv6 router openfabric 1
openfabric passive
exit
!
router openfabric 1
net 49.0000.0000.000X.00
exit
!
-
you may need to pres return after the last
!to get to a new line - if so do this -
exit the configure mode with the command
end -
save the configu with
write memory -
show the configure applied correctly with
show running-config- note the order of the items will be different to how you entered them and thats ok. (If you made a mistake i found the easiest way was to edt/etc/frr/frr.conf- but be careful if you do that.) -
use the command
exitto leave setup -
repeat steps 1 to 9 on the other 3 nodes
-
once you have configured all 3 nodes issue the command
vtysh -c "show openfabric topology"if you did everything right you will see:
Area 1:
IS-IS paths to level-2 routers that speak IP
Vertex Type Metric Next-Hop Interface Parent
pve1
10.0.0.81/32 IP internal 0 pve1(4)
pve2 TE-IS 10 pve2 en06 pve1(4)
pve3 TE-IS 10 pve3 en05 pve1(4)
10.0.0.82/32 IP TE 20 pve2 en06 pve2(4)
10.0.0.83/32 IP TE 20 pve3 en05 pve3(4)
IS-IS paths to level-2 routers that speak IPv6
Vertex Type Metric Next-Hop Interface Parent
pve1
fc00::81/128 IP6 internal 0 pve1(4)
pve2 TE-IS 10 pve2 en06 pve1(4)
pve3 TE-IS 10 pve3 en05 pve1(4)
fc00::82/128 IP6 internal 20 pve2 en06 pve2(4)
fc00::83/128 IP6 internal 20 pve3 en05 pve3(4)
IS-IS paths to level-2 routers with hop-by-hop metric
Vertex Type Metric Next-Hop Interface Parent
Now you should be in a place to ping each node from evey node across the thunderbolt mesh using IPv4 or IPv6 as you see fit.
Absolutely. Although this is highly unrecommended, it certainly works. The trick is that the UI doesn't allow you to create a monitor with a minimum of 1 node for read and write operations, but it's possible to change it later, after creation. You can create a monitor with default settings and immediately afterward adjust it to two nodes normal and 1 node minimum. It shows a warning but works.
Moreover, it's even possible to create a Ceph cluster with only one disk and add the second disk on the second node later. I did this because I had some data on one node that I wanted to move to the Ceph pool. So I created a pool with one disk first, then moved the data to the pool from the disk on the second node, and added the second disk afterward.