sergio sergiopatino

Build a Proxmox High Available cluster with Ceph

Part of collection: Hyper-converged Homelab with Proxmox

This is part 2 focussing on building the Proxmox Cluster and setting up Ceph.

See also Part 1 about Setup Networking for a High Available cluster with Ceph, and see Part 2 for how to setup the Proxmox and Ceph Cluster itself, and part 3 focussing on Managing and Troubleshooting Proxmox and Ceph.

If everything went well in part 1, setting up Proxmox and Ceph should be 'a walk in the park'!

Hyper-converged High Available Homelab with Proxmox

This is me documenting my journey moving my Homelab from a Qnap NAS and a Single host Proxmox server to a Hyper-converged multi-node Proxmox Cluster.

The reason to document it here is twofold:

Information often it scattered 'all over the place', but never 100% applicable to the setup I have.
To remember 'what the fuck' did I do some months ago.
Writing it for 'a public' forces me to think it all through again and make sure it's correct.

It's written 'first to scratch my own itch' but hopefully it benefits others too, or even better, that others improve upon my implementations. Feel free to comment or share improvements and insights!

ProxMox Cluster - Soup-to-Nutz

aka what i did to get from nothing to done.

note: these are designed to be primarily a re-install guide for myself (writing things down helps me memorize the knowledge), as such don't take any of this on blind faith - some areas are well tested and the docs are very robust, some items, less so). YMMV

Purpose of Proxmox cluster project

Required Outomces of cluster project

Disable Device Enrollment Program (DEP) notification on macOS Ventura.md

Ventura docs for M2 Macs in this comment: https://gist.github.com/henrik242/65d26a7deca30bdb9828e183809690bd?permalink_comment_id=4555340#gistcomment-4555340

Old Monterey docs in this old revision: https://gist.github.com/henrik242/65d26a7deca30bdb9828e183809690bd/32c410e3a1de73539c76fa13ea5486569c4e0c5d

Solution for Sonoma: https://gist.github.com/sghiassy/a3927405cf4ffe81242f4ecb01c382ac

Disable Device Enrollment Notification on Mac.md

Restart the Mac in Recovery Mode by holding `Comment-R` during restart

Open Terminal in the recovery screen and type

csrutil disable

	<?xml version="1.0" encoding="UTF-8"?>
	<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
	<plist version="1.0">
	<dict>
	<key>PayloadContent</key>
	<array>

	<!-- Home: Manual -->
	<dict>
	<key>UserDefinedName</key>

	Use this as an example on how to start the virtual console without the need of Java Web Start or accessing it from the web interface.
	You can use the user and password that you use for the web interface.

	You need an old JRE... I used 1.7.0_80 from the Server JRE package, also I have tested successfully 1.7.0_79 with MacOS.
	You don't need to install it, just extract it or copy the files in "jre" folder.

	Open the viewer.jnlp file that you get by launching the virtual console from the web interface with a text editor.
	Note the urls to the jar files. Download the main jar file avctKVM.jar and the libs for your operating system and architecture.
	Extract the dlls (.so Linux, .jnilib MacOS) from the jar libs.

	firewall {
	all-ping enable
	broadcast-ping disable
	ipv6-receive-redirects disable
	ipv6-src-route disable
	ip-src-route disable
	log-martians enable
	name WAN_IN {
	default-action drop
	description "WAN to internal"

	#!/usr/bin/env python

	# An exponential backoff around Boto3 DynamoDB, whose own backoff eventually
	# fails on long multipage scans. We'd like to use this as a wrapper somehow,
	# see: https://gist.github.com/numberoverzero/cec21b8ca715401c5662

	from time import sleep

	import boto3
	from boto3.dynamodb.conditions import Attr