Proxmox cluster Setup

proxmox-cluster.md

Proxmox cluster Setup

this gist is part of this series

Network Design

Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:

1. ceph shold have a dedicated network
2. proxmox should not have migration traffic and cluster communications network
3. one wants cluster communicationsnetwork reddundant

I have 3 networks:

1. Onboard 2.5gb NIC connected to one switch for subnet 192.168.1.0/24 (my LAN)

2. Thunderbolt mesh connected in a ring for subnet 10.0.0.80/28

   • this has 3 subnets 10.0.0.81/32, 10.0.0.82/32 and 10.0.0.83/32 these are used for OSPF routing between nodes

3. Addtional 2.5Gbe using (NUCIOALUWS) add-on afor subnet TBD

   • cluster (aka corosync) network uses network 1 (2.5gbe)
   • ceph migration traffic uses network 2 (thunderbolt)
   • ceph public network uses network 2 (thunderbolt
   • CT and VM migration traffic uses network 2 (thunderbolt network)

I have not yet decided what network 3 will be used for, options are:

• cluster public network that other devices use to access the cluster or its resources
• backup corosync (though i don't see a reason not to have corosync on all 3 networks)
• ceph public network - but I assume this is what the VMs uses so it makes sense to i want that on the 26Gbps thunderbolt mesh too

Create Cluster

You should have 3 browser tabs open for this, one for each node's management IP.

setup on node 1

1. navigate to Datacenter > pve1 > Cluster and click Create Cluster
2. name the cluster e.g. pve-cluster1
3. set link 0 to the IPv4 address (in my case 192.168.1.81 on interface vmbr0)
4. click create

Join node 2

1. on node 1 in Datacenter > pve1 > Cluster click join information
2. the IP address should be node 1 IPv4 address
3. click copy information
4. open tab 2 in your browser to node 2 management page
5. navingate to Datacenter > pve2 > Cluster and click join cluster
6. paste the information into the dialog box that you collected in step 3
7. Fill the root password in of node 1
8. Select Link 0 as 192.168.1.82
9. click button join 'pve-cluster1'

Join node 3

1. on node 1 in Datacenter > pve1 > Cluster click join information
2. the IP address should be node 1 IPv4 address
3. click copy information
4. open tab 2 in your browser to node 3 management page
5. navingate to Datacenter > pve3 > Cluster and click join cluster
6. paste the information into the dialog box that you collected in step 3
7. Fill the root password in of node 1
8. Select Link 0 as 192.168.1.83
9. click button join 'pve-cluster1'

at this point close your pv2 and pve 3 tabs - you can now manage all 3 cluster nodes from node 1 (or any node)

Define Migration Network

1. navigate in webui to Datacenter > Options
2. double click Migration Settings
3. select network 10.0.0.81/32 and click ok
4. edit with nano /etc/pve/datacenter.cfg and change: migration: network=10.0.0.81/32,type=secure to migration: network=10.0.0.80/29,type=insecure This is a)this subnet contains 192.168.1.81/32, 192.168.1.82/32 and 192.168.1.83/32; and b) because it is 100% isolated network it can be insecure give a small speed boost

Configuring for High Availability

1. navigate in webui to Datacenter > HA > Groups
2. click create
3. Name the cluster (ID) ClusterGroup1
4. add all 3 nodes and then click create

So the change to the subnet addresses, is that done in the /etc/network/interfaces.d/thunderbolt file as detailed in the Enable Dual Stack (IPv4 and IPv6) OpenFabric Routing, https://gist.github.com/scyto/4c664734535da122f4ab2951b22b2085 ?
I am wondering if 8.3 has changed a couple of things as I also don't see the lo:0 and lo:6 networks in the GUI.

---Update---
The only way I could get the lo:0 and lo:6 to appear was to include the following in the /etc/networking/interfaces

iface lo inet loopback

auto lo:0
iface lo:0 inet static
address 10.0.0.82/27

auto lo:6
iface lo:6 inet static
address fc00::82/64

auto en05
iface en05 inet manual
#do not edit it GUI

auto en06
iface en06 inet manual
#do not edit in GUI

iface enp87s0 inet manual
--- Snip ---
I have left the lo interfaces as given in the Enable Dual Stack in the /etc/network/interfaces.d/thunderbolt and updated the netmask. When modified in the thunderbolt file, the networks did not appear in the GUI and I was unable to change the cluster migration options to select the thunderbolt network.
Non /64 sub-net masks don't comply with the 64 bit interface address defined in RFC4291 and it makes my IPV6 OCD twitch :-)

Note for people (like me) who are not very familiar with the networking aspect:
If you use IP addresses for your thunderbolt/mesh network that are different from the examples given in this documentation and you're getting "no IP address" errors when trying to migrate VMs, you may need to recalculate the CIDR subnet mask. This documentation uses 10.0.0.81-83, but I wanted to use .31-33 (to match the external IPv4 addresses of the hosts), but when I configured datacenter.cfg to use 10.0.0.30/29, I got the error "proxmox could not get migration ip: no IP address configured on local node for network" when trying to migrate VMs.

After a bunch of digging, I realized that *.30/29 gives a usable IP address range of .25-.30... excluding the addresses I was using. So I tweaked the values in an IP Subnet Calculator until I landed on *.30/26, which gives a usable range of 10.0.0.1-10.0.0.62, which includes all of the IP addresses I'm using. Migrations are now happening fast and flawlessly.