this gist is part of this series
Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:
- ceph shold have a dedicated network
- proxmox should not have migration traffic and cluster communications network
- one wants cluster communicationsnetwork reddundant
I have 3 networks:
-
Onboard 2.5gb NIC connected to one switch for subnet 192.168.1.0/24 (my LAN)
-
Thunderbolt mesh connected in a ring for subnet 10.0.0.80/28
- this has 3 subnets 10.0.0.81/32, 10.0.0.82/32 and 10.0.0.83/32 these are used for OSPF routing between nodes
-
Addtional 2.5Gbe using (NUCIOALUWS) add-on afor subnet TBD
- cluster (aka corosync) network uses network 1 (2.5gbe)
- ceph migration traffic uses network 2 (thunderbolt)
- ceph public network uses network 2 (thunderbolt
- CT and VM migration traffic uses network 2 (thunderbolt network)
I have not yet decided what network 3 will be used for, options are:
- cluster public network that other devices use to access the cluster or its resources
- backup corosync (though i don't see a reason not to have corosync on all 3 networks)
- ceph public network - but I assume this is what the VMs uses so it makes sense to i want that on the 26Gbps thunderbolt mesh too
You should have 3 browser tabs open for this, one for each node's management IP.
- navigate to
Datacenter > pve1 > Clusterand clickCreate Cluster - name the cluster e.g.
pve-cluster1 - set link 0 to the IPv4 address (in my case
192.168.1.81on interface vmbr0) - click
create
- on node 1 in
Datacenter > pve1 > Clusterclickjoin information - the IP address should be node 1 IPv4 address
- click
copy information - open tab 2 in your browser to node 2 management page
- navingate to
Datacenter > pve2 > Clusterand click join cluster - paste the information into the dialog box that you collected in step 3
- Fill the root password in of node 1
- Select Link 0 as
192.168.1.82 - click button
join 'pve-cluster1'
- on node 1 in
Datacenter > pve1 > Clusterclickjoin information - the IP address should be node 1 IPv4 address
- click
copy information - open tab 2 in your browser to node 3 management page
- navingate to
Datacenter > pve3 > Clusterand click join cluster - paste the information into the dialog box that you collected in step 3
- Fill the root password in of node 1
- Select Link 0 as
192.168.1.83 - click button
join 'pve-cluster1'
at this point close your pv2 and pve 3 tabs - you can now manage all 3 cluster nodes from node 1 (or any node)
- navigate in webui to
Datacenter > Options - double click
Migration Settings - select network
10.0.0.81/32and click ok - edit with
nano /etc/pve/datacenter.cfgand change:migration: network=10.0.0.81/32,type=securetomigration: network=10.0.0.80/29,type=insecureThis is a)this subnet contains 192.168.1.81/32, 192.168.1.82/32 and 192.168.1.83/32; and b) because it is 100% isolated network it can be insecure give a small speed boost
- navigate in webui to
Datacenter > HA > Groups - click create
- Name the cluster (ID)
ClusterGroup1 - add all 3 nodes and then click
create
When following the updated guide to create a separate folder for the thunderbolt networking "/etc/network/interfaces.d/thunderbolt" the interfaces lo:0 and lo:6 don't show up in the GUI to allow for setting the migration network. I manually edited the /etc/pve/datacenter.cfg with "migration: network=fc00::81/128,type=insecure" and it shows in the GUI as changed but I haven't tested migration yet. Not sure if this is correct or if I messed something up which isn't letting me select the network via GUI.