Skip to content

Instantly share code, notes, and snippets.

@kafisatz

kafisatz/proxmox-cluster.md

Forked from scyto/proxmox-cluster.md
Created September 15, 2023 18:11
Show Gist options
  • Save kafisatz/9061e55d3ce24c1a54170e054860fa9f to your computer and use it in GitHub Desktop.
Save kafisatz/9061e55d3ce24c1a54170e054860fa9f to your computer and use it in GitHub Desktop.
Download ZIP
Proxmox cluster Setup
Raw
proxmox-cluster.md

Proxmox cluster Setup

Network Design

Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:

  1. ceph shold have a dedicated network
  2. proxmox should not have migration traffic and cluster communications network
  3. one wants cluster communicationsnetwork reddundant

I have 3 networks:

  1. Onboard 2.5gb NIC connected to one switch for subnet 192.168.1.0/24 (my LAN)

  2. Thunderbolt mesh connected in a ring for subnet 10.0.0.80/28

    • this has 3 subnets 10.0.0.81/32, 10.0.0.82/32 and 10.0.0.83/32 these are used for OSPF routing between nodes

  3. Addtional 2.5Gbe using (NUCIOALUWS) add-on afor subnet TBD

    • cluster (aka corosync) network uses network 1 (2.5gbe)
    • ceph migration traffic uses network 2 (thunderbolt)
    • ceph public network uses network 2 (thunderbolt
    • CT and VM migration traffic uses network 2 (thunderbolt network)

I have not yet decided what network 3 will be used for, options are:

  • cluster public network that other devices use to access the cluster or its resources
  • backup corosync (though i don't see a reason not to have corosync on all 3 networks)
  • ceph public network - but I assume this is what the VMs uses so it makes sense to i want that on the 26Gbps thunderbolt mesh too

Create Cluster

You should have 3 browser tabs open for this, one for each node's management IP.

setup on node 1

  1. navigate to Datacenter > pve1 > Cluster and click Create Cluster
  2. name the cluster e.g. pve-cluster1
  3. set link 0 to the IPv4 address (in my case 192.168.1.81 on interface vmbr0)
  4. click create

Join node 2

  1. on node 1 in Datacenter > pve1 > Cluster click join information
  2. the IP address should be node 1 IPv4 address
  3. click copy information
  4. open tab 2 in your browser to node 2 management page
  5. navingate to Datacenter > pve2 > Cluster and click join cluster
  6. paste the information into the dialog box that you collected in step 3
  7. Fill the root password in of node 1
  8. Select Link 0 as 192.168.1.82
  9. click button join 'pve-cluster1'

Join node 3

  1. on node 1 in Datacenter > pve1 > Cluster click join information
  2. the IP address should be node 1 IPv4 address
  3. click copy information
  4. open tab 2 in your browser to node 3 management page
  5. navingate to Datacenter > pve3 > Cluster and click join cluster
  6. paste the information into the dialog box that you collected in step 3
  7. Fill the root password in of node 1
  8. Select Link 0 as 192.168.1.83
  9. click button join 'pve-cluster1'

at this point close your pv2 and pve 3 tabs - you can now manage all 3 cluster nodes from node 1 (or any node)

Define Migration Network

  1. navigate in webui to Datacenter > Options
  2. double click Migration Settings
  3. select network 10.0.0.81/32 and click ok
  4. edit with nano /etc/pve/datacenter.cfg and change: migration: network=10.0.0.81/32,type=secure to migration: network=10.0.0.80/29,type=insecure This is a)this subnet contains 192.168.1.81/32, 192.168.1.82/32 and 192.168.1.83/32; and b) because it is 100% isolated network it can be insecure give a small speed boost

Configuring for High Availability

  1. navigate in webui to Datacenter > HA > Groups
  2. click create
  3. Name the cluster (ID) ClusterGroup1
  4. add all 3 nodes and then click create
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment