Skip to content

Instantly share code, notes, and snippets.

@scyto
Created April 28, 2025 04:45
Show Gist options
  • Save scyto/a02bbcf947f4a18773c30fa3d12bf495 to your computer and use it in GitHub Desktop.
Save scyto/a02bbcf947f4a18773c30fa3d12bf495 to your computer and use it in GitHub Desktop.

Thanks for hanging tight β€”
Here’s the corrected, production-quality version based on everything you asked:


πŸ›  Thunderbolt Mesh Setup – Staged Guide (Proxmox + FRR + BGP)


πŸ“¦ Stage 1 β€” Internal Mesh VM Routing Only

In this stage you will:

  • Build the routed Thunderbolt mesh
  • Set up OpenFabric between Proxmox nodes
  • Route VMs between nodes
  • No LAN / external routing yet

1. Prepare Network Interfaces

(unchanged from before, already correct)


2. Install and Configure FRRouting

(unchanged from before, already correct)


3. Configure /etc/frr/frr.conf for Mesh Only (No LAN Yet)

frr version 8.5.2
frr defaults datacenter
hostname pve1
log syslog informational
service integrated-vtysh-config

interface en05
 ip router openfabric 1
 ipv6 router openfabric 1
 mtu 65520

interface en06
 ip router openfabric 1
 ipv6 router openfabric 1
 mtu 65520

interface lo
 ip router openfabric 1
 ipv6 router openfabric 1
 openfabric passive

interface vmbr100
 ipv6 router openfabric 1
 openfabric passive

router openfabric 1
 net 49.0000.0000.0081.00
 lsp-gen-interval 5

4. Deploy and Verify OpenFabric

Apply networking:

ifreload -a
systemctl restart frr

Check OpenFabric IS-IS neighbors:

vtysh -c "show isis neighbor"

Example output:

System ID      Interface  State  Priority Circuit ID
pve2           en05       Up     64       0
pve3           en06       Up     64       0

βœ… Neighbors between nodes are UP.


🌍 Stage 2 β€” Add LAN Access via Unifi Router


5. Extend FRR Config for BGP (with Explicit Networks)

Append to /etc/frr/frr.conf:

router bgp 65001
 bgp router-id 192.168.1.81
 no bgp ebgp-requires-policy

 neighbor 192.168.1.1 remote-as 65001
 neighbor 192.168.1.1 update-source 192.168.1.81
 neighbor 192.168.1.1 next-hop-self

 neighbor 2001:db8:1000:1::1 remote-as 65001
 neighbor 2001:db8:1000:1::1 update-source 2001:db8:1000:1::81
 neighbor 2001:db8:1000:1::1 next-hop-self

 address-family ipv6 unicast
  network 2001:db8:1000:1::81/128
  network fc00:81::/64
  network fc00::81/128
  neighbor 2001:db8:1000:1::1 activate
  neighbor 2001:db8:1000:1::1 next-hop-self
 exit-address-family

 address-family ipv4 unicast
  network 10.0.0.81/32
  network 10.0.81.0/24
  neighbor 192.168.1.1 activate
  neighbor 192.168.1.1 next-hop-self
 exit-address-family

βœ… No redistribute connected / redistribute kernel anymore.
βœ… Only clean, precise prefixes are advertised.


6. Configure BGP on Unifi Router

router bgp 65001
 bgp router-id 192.168.1.1
 no bgp ebgp-requires-policy

 neighbor 192.168.1.81 remote-as 65001
 neighbor 192.168.1.81 update-source 192.168.1.1
 neighbor 192.168.1.81 next-hop-self

 neighbor 192.168.1.82 remote-as 65001
 neighbor 192.168.1.82 update-source 192.168.1.1
 neighbor 192.168.1.82 next-hop-self

 neighbor 192.168.1.83 remote-as 65001
 neighbor 192.168.1.83 update-source 192.168.1.1
 neighbor 192.168.1.83 next-hop-self

 neighbor 2001:db8:1000:1::81 remote-as 65001
 neighbor 2001:db8:1000:1::81 update-source 2001:db8:1000:1::1
 neighbor 2001:db8:1000:1::81 next-hop-self

 neighbor 2001:db8:1000:1::82 remote-as 65001
 neighbor 2001:db8:1000:1::82 update-source 2001:db8:1000:1::1
 neighbor 2001:db8:1000:1::82 next-hop-self

 neighbor 2001:db8:1000:1::83 remote-as 65001
 neighbor 2001:db8:1000:1::83 update-source 2001:db8:1000:1::1
 neighbor 2001:db8:1000:1::83 next-hop-self

 address-family ipv6 unicast
  redistribute connected
  redistribute kernel
 exit-address-family

 address-family ipv4 unicast
  redistribute connected
  redistribute kernel
 exit-address-family

βœ… The Unifi router can still redistribute because it aggregates multiple mesh nodes.


7. Validate Full Mesh + LAN Access

Check BGP neighbor status:

vtysh -c "show bgp summary"

Check IPv6 routing table:

vtysh -c "show bgp ipv6 unicast"

Example updated output:

   Network                   Next Hop
*> 2001:db8:1000:1::81/128    fe80::dead:beef:1
*> 2001:db8:1000:1::82/128    fe80::dead:beef:2
*> 2001:db8:1000:1::83/128    fe80::dead:beef:3
*> fc00:81::/64               fe80::dead:beef:1
*> fc00:82::/64               fe80::dead:beef:2
*> fc00:83::/64               fe80::dead:beef:3

βœ… IPv6 advertised correctly.

Check IPv4 routing table:

vtysh -c "show bgp ipv4 unicast"

Example output:

   Network          Next Hop
*>i10.0.0.81/32     192.168.1.81
*>i10.0.0.82/32     192.168.1.82
*>i10.0.0.83/32     192.168.1.83
*>i10.0.81.0/24     192.168.1.81
*>i10.0.82.0/24     192.168.1.82
*>i10.0.83.0/24     192.168.1.83

βœ… IPv4 advertised correctly.


πŸ“‹ Network Advertisement Table (Corrected)

Node External IPv6 (loopback) VM Subnet IPv6 External IPv4 (loopback) VM Subnet IPv4
pve1 2001:db8:1000:1::81/128 fc00:81::/64 10.0.0.81/32 10.0.81.0/24
pve2 2001:db8:1000:1::82/128 fc00:82::/64 10.0.0.82/32 10.0.82.0/24
pve3 2001:db8:1000:1::83/128 fc00:83::/64 10.0.0.83/32 10.0.83.0/24

🎯 Conclusion

βœ… Stage 1 = Internal routed VM Thunderbolt mesh
βœ… Stage 2 = External LAN reachability via Unifi BGP integration
βœ… Clean explicit advertisement, no risky redistribution


πŸš€ Final Step:

Would you like me to immediately:

  • Bundle this corrected text into a clean .md file?
  • Also generate an optional fancy version with a table of contents?

(It’s ready if you want me to!) πŸ“„πŸš€
Just say yes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment