Thunderbolt Mesh Cluster Setup Guide (Proxmox + FRR + BGP)

📁 Table of Contents

1. Network Overview
2. Interface Overview (All Nodes)
3. Stage 1 — Internal Mesh VM Routing Only
4. Stage 2 — LAN Access via Unifi Router
5. Conclusion

1. Network Overview

Three Proxmox nodes interconnected via Thunderbolt.
Routed point-to-point interfaces, no bridging.
FRRouting (FRR) with OpenFabric (IS-IS) for internal mesh routing.
BGP to Unifi router for external LAN access.
All services explicitly advertised via network statements (no redistribute connected).

2. Interface Overview (All Nodes)

Node	Interface	Address (CIDR)	MTU	Purpose
pve1	vmbr0	192.168.1.81/24	9182	Management IPv4
pve1	vmbr0	2001:db8:1000:1::81/64	9182	Management IPv6
pve1	en05	(no IP)	65520	Thunderbolt Fabric
pve1	en06	(no IP)	65520	Thunderbolt Fabric
pve1	vmbr100	10.0.81.1/24	65520	VM Routed IPv4
pve1	vmbr100	fc00:81::1/64	65520	VM Routed IPv6
pve1	lo	10.0.0.81/32	virtual	Loopback IPv4
pve1	lo	fc00::81/128	virtual	Loopback IPv6

(pve2 and pve3 follow similarly with .82/.83 addresses)

3. Stage 1 — Internal Mesh VM Routing Only

3.1 Configure Network Interfaces

Edit /etc/network/interfaces and /etc/network/interfaces.d/thunderbolt to match required MTUs and static assignments for Thunderbolt and loopbacks.

3.2 Install FRRouting - not needed on promox 8.4+

apt update
apt install frr
systemctl enable frr

3.3 Configure OpenFabric

Example /etc/frr/frr.conf:

frr version 8.5.2
frr defaults datacenter
hostname pve1
log syslog informational
service integrated-vtysh-config

interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
exit

interface en06
 ip router openfabric 1
 ipv6 router openfabric 1
exit

interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive
exit

interface vmbr100
 ipv6 router openfabric 1
 openfabric passive
exit

router openfabric 1
 net 49.0000.0000.0081.00
 lsp-gen-interval 5
exit

3.4 Verify Internal Mesh Connectivity

vtysh -c "show openfabric topology"

4. Stage 2 — LAN Access via Unifi Router

4.1 Extend FRR for BGP

Example /etc/frr/frr.conf extension:

router bgp 65001
 bgp router-id 192.168.1.81
 no bgp ebgp-requires-policy

 neighbor 192.168.1.1 remote-as 65001
 neighbor 192.168.1.1 update-source 192.168.1.81
 neighbor 192.168.1.1 next-hop-self

 neighbor 2001:db8:1000:1::1 remote-as 65001
 neighbor 2001:db8:1000:1::1 update-source 2001:db8:1000:1::81
 neighbor 2001:db8:1000:1::1 next-hop-self

 address-family ipv6 unicast
  network 2001:db8:1000:1::81/128
  network fc00::81/128
  network fc00:81::/64
  neighbor 2001:db8:1000:1::1 activate
  neighbor 2001:db8:1000:1::1 next-hop-self
 exit-address-family

 address-family ipv4 unicast
  network 10.0.0.81/32
  network 10.0.81.0/24
  neighbor 192.168.1.1 activate
  neighbor 192.168.1.1 next-hop-self
 exit-address-family

4.2 Configure Unifi Router BGP

router bgp 65001
 bgp router-id 192.168.1.1
 no bgp ebgp-requires-policy

 neighbor 192.168.1.81 remote-as 65001
 neighbor 192.168.1.81 update-source 192.168.1.1
 neighbor 192.168.1.81 next-hop-self
 (repeat for .82, .83)

 neighbor 2001:db8:1000:1::81 remote-as 65001
 neighbor 2001:db8:1000:1::81 update-source 2001:db8:1000:1::1
 neighbor 2001:db8:1000:1::81 next-hop-self
 (repeat for ::82, ::83)

 address-family ipv6 unicast
  redistribute connected
  redistribute kernel
 exit-address-family

 address-family ipv4 unicast
  redistribute connected
  redistribute kernel
 exit-address-family

4.3 Validate BGP Route Exchanges

vtysh -c "show bgp summary"
vtysh -c "show bgp ipv6 unicast"
vtysh -c "show bgp ipv4 unicast"

Example IPv6 table:

*> 2001:db8:1000:1::81/128    fe80::dead:beef:1
*> 2001:db8:1000:1::82/128    fe80::dead:beef:2
*> 2001:db8:1000:1::83/128    fe80::dead:beef:3
*> fc00:81::/64               fe80::dead:beef:1
*> fc00:82::/64               fe80::dead:beef:2
*> fc00:83::/64               fe80::dead:beef:3

Example IPv4 table:

*>i10.0.0.81/32     192.168.1.81
*>i10.0.0.82/32     192.168.1.82
*>i10.0.0.83/32     192.168.1.83
*>i10.0.81.0/24     192.168.1.81
*>i10.0.82.0/24     192.168.1.82
*>i10.0.83.0/24     192.168.1.83

5. Conclusion

✅ Internal Thunderbolt Mesh operational
✅ External LAN reachability via Unifi router
✅ No redistribute connected on Proxmox nodes
✅ Safe for production with documentation-compliant IPv6 examples

scyto/refactor2.md