free-pmx

Proxmox VE - Backup Cluster config (pmxcfs) - /etc/pve

Backup

A no-nonsense way to safely backup your /etc/pve files (pmxcfs) [^CFS] is actually very simple:

sqlite3 /var/lib/pve-cluster/config.db .dump > ~/config.dump.$(date --utc +%Z%Y%m%d%H%M%S).sql

free-pmx

Why Proxmox VE shreds your SSDs

You must have read, at least once, that Proxmox recommend "enterprise" SSDs for their virtualisation stack. But why does it shred regular SSDs? It would not have to, in fact the modern ones, even without PLP, can endure as much as 2,000 TBW per life. But where do the writes come from? ZFS? Let's have a look.

The below is particularly of interest for any homelab user, but in fact everyone who cares about wasted system performance might be interested.

If you have a cluster, you can actually safely follow this experiment. Add a new "probe" node that you will later dispose of and let it join the cluster. On the "probe" node, let's isolate the configuration state backend database onto a separate filesystem, to be able to benchmark only pmxcfs - the virtual filesystem that is mounted to /etc/pve and holds your configuration files, i.e. cluster state.

dd if=/dev/zero of=/root/pmxcfsbd bs=1M count=256

free-pmx

Proxmox VE - Misdiagnosed: failed to load local private key

If you encounter this error in your logs, your GUI is also inaccessible. You would have found it with console access or direct SSH:

journalctl -e

This output will contain copious amount of:

free-pmx

Passwordless SSH can lock you out of a node

If you follow standard security practices, you would not allow root logins, let alone connections over SSH (as with Debian standard install). But this would deem your PVE unable to function properly, so you can only resort to fix your /etc/ssh/sshd_config [^SSHDC] with the option:

PermitRootLogin prohibit-password

That way, you only allow connections with valid keys (not password). Prior to this, you would have copied over your public keys with ssh-copy-id [^SSHCI] or otherwise add them to /root/.ssh/authorized_keys.

free-pmx

Proxmox VE - DHCP Deployment

---

DISCLAIMER You WILL suffer from the same issues as with static network configuration should your node IP or hostname change in terms of managing the transition. While it actually is possible to change both without a reboot (more on that below), the intended use case is to cover a rather stable environment, but allow for centralised management.

---

PVE static network configuration[^IFCS] is not actually a real prerequisite, not even for clusters.

*** Tested with PVE 8.2. ***

free-pmx

The improved SSH with hidden regressions

If you pop into the release notes of PVE 8.2, [^RN] there's a humble note on changes to SSH behaviour under Improved management for Proxmox VE clusters:

Modernize handling of host keys for SSH connections between cluster nodes ([bugreport] 4886).

Previously, /etc/ssh/ssh_known_hosts was a symlink to a shared file containing all node hostkeys. This could cause problems if conflicting hostkeys appeared in /root/.ssh/known_hosts, for example after re-joining a node to the cluster under its old name. Now, each node advertises its own host key over the cluster filesystem. When Proxmox VE initiates an SSH connection from one node to another, it pins the advertised host key. For existing clusters, pvecm updatecerts can optionally unmerge the existing /etc/ssh/ssh_known_hosts.

free-pmx

Safely disable HA auto-reboots for maintenance

If you are going to perform any kind of maintenance works which could disrupt your quorum cluster-wide (e.g. network equipment, small clusters), you would have learnt this risks seemingly random reboots on cluster nodes with (not only) active HA services. [^HAF]

To safely disable HA without additional waiting times and avoiding long-term bugs (which Proxmox do not care for [^BZ5243]), you will want to perform the following:

Before the works

Once (on any node):

free-pmx

Within /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js,
there's a function checked_command: function(orig_cmd),
inside of which:

--- proxmoxlib.js
+++ proxmoxlib.no-nag.js

         success: function(response, opts) {
- let res = response.result;