kafisatz

kafisatz

#see also https://discourse.julialang.org/t/predefined-dates-locale/81250
#generated with openai.com

#an arbitrary selection of languages for date locales

@warn("Use with caution. The words and abbreviations have not been reviewed!")
using Dates 

let 
    months = ["Januar", "Februar", "März", "April", "Mai", "Juni", "Juli", "August", "September", "Oktober", "November", "Dezember"]

kafisatz

Thunderbolt Networking

Install LLDP - this is great to see what nodes can see which.

• install lldpctl with apt install lldpd

Load Kernel Modules

• add thunderbolt and thunderbolt-net kernel modules (this must be done all nodes - yes i know it can sometimes work withoutm but the thuderbolt-net one has interesting behaviou' so do as i say - add both ;-)
  1. nano /etc/modules add modules at bottom of file, one on each line
  2. save using x then y then enter

kafisatz

ProxMox Cluster Proof Of Concept (Work In Progress)

Start Date - First week of August 2023

End Date - none

Last Updated 9/1/2023

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust (some of the extra credit items, less so). YMMV i wiill regularly update until this become production

kafisatz

Hardware Setup / Proxmox Base Install

Hardware Setup (perform on all 3 nodes)

• Connect all 3 NUC to power etc
• Disable secure boot in BIOS on all nodes
• Connect all 3 TB4 cables in a rinng topology:

using the numbers printed on the case of the intel13 nucs connect cables as follows (this is important):
- node 1 port 1 > node 2 port 2
- node 2 port 1 > node 3 port 2

kafisatz

Enable IPv4 OSPF Routing on Thunderbolt Mesh

This will result in a routable mesh network that can survive any one node failure or any one cable failure. Alls the steps in this section must be performed on each node

Key Parameters

Key Information Used Note i used the 10.x IPv4 space as this is not used anywhere else on my network YMMV

Node l:

• lo:0 = 10.0.0.81/32

kafisatz

Proxmox cluster Setup

Network Design

Put simply I am not sure what the design should be. I have the thunderbolt mesh network and the 2.5gbe NIC on each node. The ideal design guidelies cause my brain to have a race conditions because:

1. ceph shold have a dedicated network
2. proxmox should not have migration traffic and cluster communications network
3. one wants cluster communicationsnetwork reddundant

I have 3 networks:

kafisatz

CEPH HA Setup

Note this should only be done once you are sure you have reliable network. proxmox and ceph seem fragile wrt to changing underlying network after configuration of ceph or even during all installation done via command line due to gui not understanding the mesh network

This setup doesn't attempt to seperate the ceph public network and ceph cluster network (not same as proxmox clutser network), The goal is to get an easy working setup.

Ceph Initial Install & monitor creation

1. On all nodes execute the command pveceph install --repository no-subscription accept all the packages and install
2. On node 1 execute the command pveceph init --network 10.0.0.81/24

kafisatz

Configuring CephFS to store ISOs for VMs and Templates for Containers

This gist assume you have setup Ceph as per earlier gist in this sequence

I found it annoying to upload ISO to each node in the cluster, download CT templates to each node.
I thought setting up cedph for these would be a great idea (and it is).

create cephfs pools

1. In the gui navigate to Datcenter > node name > ceph
2. Click cephfs node
3. In Metadata Server area click create and create a meta data server on all 3 pve nodes

kafisatz

Setup HA Windows VM

this example uses windows server 2022. Windows 11 will be basically the same

It is beyond this gist to be a deep tutorial.

Prep

1. Download your windows ISO of choice (i am using windows 2022 Data Center for this example.)
2. Download the lates virtio.iso too
3. go to Datacenter > pve1 > expand child nodes in left pane > local (pve1) (i don't know why i find this unintutive)
4. click ISO Images