Passwordless SSH can lock you out of a node

If you follow standard security practices, you would not allow root logins, let alone connections over SSH (as with Debian standard install). But this would deem your PVE unable to function properly, so you can only resort to fix your /etc/ssh/sshd_config ¹ with the option:

PermitRootLogin prohibit-password

That way, you only allow connections with valid keys (not password). Prior to this, you would have copied over your public keys with ssh-copy-id ² or otherwise add them to /root/.ssh/authorized_keys.

But this has a huge caveat on any standard PVE install. When you examine the file, it is actually a symbolic link:

/root/.ssh/authorized_keys -> /etc/pve/priv/authorized_keys

This is because there's already other nodes' keys there to allow for cross-connecting - and the location is shared. This has several issues, most important of which is that the actual file lies in /etc/pve which is a virtual filesystem ³ mounted only when all goes well during boot-up.

What could go wrong

If your /etc/pve does not get mounted during bootup, your node will appear offline and will not be accessible over SSH, let alone GUI.

NOTE If accessing via other node's GUI, you will get confusing Permission denied (publickey,password) in the "Shell".

You are essentially locked-out, despite the system otherwise booted up except for PVE services. You cannot troubleshoot over SSH, you would need to resort to OOB management or physical access.

This is because during your SSH connection, there's no way to verify your key against the /etc/pve/priv/authorized_keys.

NOTE If you allow root to authenticate also by password, it will lock you out of "GUI only". Your SSH will not work - obviously - with key, but fallback to password prompt.

How to avoid this

You need to use your own authorized_keys, different from the default that has been hijacked by PVE. The proper way to do this is define its location in the config:

cat > /etc/ssh/sshd_config.d/LocalAuthorizedKeys.conf <<< "AuthorizedKeysFile .ssh/local_authorized_keys"

If you now copy your own keys to /root/.ssh/local_authorized_keys file (on every node), you are immune from this design flaw.

NOTE There are even better ways to approach this, e.g. SSH certificates, in which case you are not prone to encounter this bug for your own setup. This is out of scope for this post.

NOTE All respective bugs mentioned above filed with Proxmox.

free-pmx/pve-ssh-lockout.md

Passwordless SSH can lock you out of a node

What could go wrong

How to avoid this

Footnotes