Bryan Wyatt brwyatt

Diskless Desktop off Ceph RBD root

History/Background

This is the result of a project to PXE-boot a diskless desktop backed by a Ceph cluster. Initially, the plan was to use CephFS for both / (root) and /boot, with /boot accessible by the PXE server to be able to streamline updates to the initrd and kernel from the OS. While /boot is still on CephFS, / was moved to an RBD image, using namespaces for permissions.

Not covered

Setting up Ceph (CephFS, RBD, permissions/auth, etc), outside of where it directly interacts

Create a Render LXC in Proxmox

Instructions to setup a dedicated rendering LXC for Jellyfin. Useful for running Jellyfin in a VM and without passing through the whole iGPU, instead passing off to LXCs, allowing using more than one Proxmox host in the cluster for rendering.

Sources

Jellyfin install: https://linuxcapable.com/how-to-install-jellyfin-media-server-on-ubuntu-linux/
Jellyfin Intel GPU config: https://jellyfin.org/docs/general/post-install/transcoding/hardware-acceleration/intel#lxc-and-lxd-container
Remote ffmpeg: https://github.com/joshuaboniface/rffmpeg?tab=readme-ov-file

Hardware

Minisforum MS-01

Workaround for missing default route on Proxmox boot

Have an issue where Proxmox, on boot, fails to set the default route. It is set in the config (file and UI), and changing the gateway and applying will set the default route, so something's getting mixed up here. So hopefully this fixes it.

Replace 172.17.25.1 with your gateway, and vlan25 with the correct interface.

Script

file: /usr/local/bin/fix-default-route.sh

Installing HAProxy with Certbot

Largely taken from this tutorial, which was hilariously out of date

Install HA Proxy

sudo apt install haproxy

	#!/bin/bash

	ip=""
	ping="NO"
	dns=()
	timeout=1

	while [[ $# -gt 0 ]]; do
	case $1 in
	-p\|--ping)

	#!/bin/bash

	models=(
	"switch-enterprise-8-poe"
	"usw-enterprise-24-poe"
	"usw-enterprise-48-poe"
	"usp-rps"
	)

	do_check () {

	#!/bin/bash

	file="/tmp/count.csv"

	echo "datetime,commit,lines" > "${file}"

	for i in $(git log --pretty='%ct:%H'); do
	d=$(echo "${i}"\|cut -f1 -d':')
	c=$(echo "${i}"\|cut -f2 -d':')

	#!/bin/bash

	# Put the following in the `~/.ssh/config` for each host to
	# conditionally proxy:
	# ProxyCommand ssh_proxy.sh %h %p bastion.example.net

	if [ $# -ne 3 ]; then
	echo "USAGE: $0 HOST PORT PROXYHOST" >&2
	exit 99
	fi

	#!/bin/bash

	servers=()
	unset input
	read -p "Server ${#servers[@]}: " input
	while [ ${#servers[@]} -eq 0 ] \|\| [[ "x${input}" != "x" ]]; do
	if [[ "x${input}" != "x" ]]; then
	servers=("${servers[@]}" $input)
	fi
	unset input

	#!/bin/bash

	osds=($(systemctl list-units \| grep -iE 'ceph-osd@[[:digit:]]+\.service' \| tr -s ' ' \| cut -f2 -d' ' \| cut -f2 -d'@' \| cut -f1 -d'.'))

	for i in "${osds[@]}"; do
	echo "***Outing OSD $i"
	ceph osd out $i
	done

	read -p "***Pausing for replication/recovery. Press [ENTER] when done..." $DONE