scyto

ProxMox Cluster - Soup-to-Nutz

aka what i did to get from nothing to done.

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust, some items, less so). YMMV

Purpose of Proxmox cluster project

Required Outomces of cluster project

scyto

Hardware Setup / Proxmox Base Install

this gist is part of this series

Hardware Setup (perform on all 3 nodes)

• Connect all 3 NUC to power etc
• Disable secure boot in BIOS on all nodes
• Connect all 3 TB4 cables in a rinng topology:

using the numbers printed on the case of the intel13 nucs connect cables as follows (this is important):

scyto

Thunderbolt Networking

this gist is part of this series

you wil need proxmox kernel 6.2.16-14-pve or higher.

Load Kernel Modules

• add thunderbolt and thunderbolt-net kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-)
  1. nano /etc/modules add modules at bottom of file, one on each line

2. save using x then y then enter

scyto

Enable IPv6 OSPF Routing on Thunderbolt-Net Mesh

This requires proxmox kernel 6.2.16-14-pve or higher due to bugs in earlier version.s

This will result in a routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node

IPv6 OSPF connectivity over TB cluster network

Enable IPv6 forwarding

Using IPv6 to take advantage of not needing to use addresses - does make things simpler

scyto

Proxmox cluster Setup

this gist is part of this series

Network Design

Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:

1. ceph shold have a dedicated network
2. proxmox should not have migration traffic and cluster communications network
3. one wants cluster communicationsnetwork reddundant

scyto

Enable IPv6 and IPv4

NOTE: as of 9/14/2023 this will only work with a custom patched linux kernel6.5.2 Proxmox have indicated they will back port these patches contact me on the proxmox forum if you wish to roll the patches yourself

Key Parameters

Nodel:

• IPv4 Address = 10.0.0.81/32
• IPv6 Address = fc00::81/128

scyto

Enable OSPF Routing on Thunderbolt Mesh

This has been deprectaed

It is now superceded by Openfabric Routing see here

continue at your own peril, for reference only now.

Old Gist

This will result in an IPv4 routable mesh network that can survive any one node failure or any one cable failure. All the steps in this section must be performed on each node

scyto

CEPH HA Setup

Note this should only be done once you are sure you have reliable TB mesh network.

this is because proxmox UI seems fragile wrt to changing underlying network after configuration of ceph.

All installation done via command line due to gui not understanding the mesh network

This setup doesn't attempt to seperate the ceph public network and ceph cluster network (not same as proxmox clutser network), The goal is to get an easy working setup.

**2025.04.24 NOTE: some folks had to switch to IPv6 for ceph due to IPv4 unreliability issues, we think as of pve 8.4.1 and all the input the community has give to update this set of gsists - that IPv4 is now reliable even on MS-01. As such i advising everyone to use IPv4 for ceph as if you have IPv6 you will have issues with SDN at this time (if you don't use SDN this is not an issue).

scyto

Setup HA Windows VM

this example uses windows server 2022. Windows 11 will be basically the same

It is beyond this gist to be a deep tutorial.

this gist is part of this series

Prep

1. Download your windows ISO of choice (i am using windows 2022 Data Center for this example.)
2. Download the lates virtio.iso too

scyto

this gist is part of this series

PROXMOX Setup Weirdness Observed

1. Most of the time gui setup didn't get dhcp address from my server, one time in 6 installs it did.
2. Sometimes gui setup populated an example IPv4 address, sometimes an actual IPv6 address and one time real DCHP IPv4 address
3. on one node in 6 installs it added a thunderbol0 entry to /etc/network/interaces resulting in what you see below, the fix was to remove from interfaces file

4. sometimes after setup the tunderbolt modules load and sometimes they don't - i don't know why, i currently have 1 machine with them specified in /etc/modules file and 2 machines without - i figured out this is (stupidly) by design, one machine in the cluster needs to manually have the the modules loaded (but this will cause other machines to load them on the fly) as such its