Proxmox VE tips

Just some tips I gathered over time. All in one easily reachable place so I can share it wherever I want.
Please note that unless you see a shebang (#!/...) these code blocks are meant to be copy & pasted.
Some of the steps will not work if you run part of them in a script and copy paste other ones.

Proxmox VE tips

Discard

Using trim/discard with thinly allocated disks (which is the default) gives space back to the storage. This saves space, makes backups faster and is needed for thin allocation to work as expected. This is not related to the PVE storage being backed by a SSD. Use it whenever the storage is thin provisioned. For ZFS this still counts even if Thin Provision (see note below) is not enabled.
Check lvs's Data% column and zfs list's USED/REFER. You might find it to go down when triggering a trim as explained below.

Also see official docs:

If you use ZFS you might also want to enable Thin Provisioning in Datacenter > Storage for your ZFS storage.
.
This will only affect newly created disks. Here's how to apply the setting for already existing disks.

CT

Containers usually cannot call fstrim themselves. You can trigger a one time immediate trim via pct fstrim IDOFCTHERE on the node.
I use a cronjob calling pct fstrim.

30 0 * * 0 pct list | awk '/^[0-9]/ {print $1}' | while read ct; do pct fstrim ${ct}; done

Alternatively you can select discard (8.3.x+) as mount option so this happens immediately.

You do not need to enable this for pct fstrim to work.
Use the mount option when you want it to be immediate/continuous and the pct fstrim cronjob to trigger it on a schedule like it usually works for VMs. I prefer the latter.

VM

You can trigger a one time immediate trim via fstrim -av from inside a VM.
Most OSs come with a fstrim.timer which, by default, does a weekly fstrim call.
You can check with systemctl status fstrim.timer. If disabled run systemctl enable fstrim.timer.
To edit it to happen more frequently run systemctl edit fstrim.timer.

Some guest operating systems may also require the SSD Emulation flag to be set. If you would like a drive to be presented to the guest as a solid-state drive rather than a rotational hard disk, you can set the SSD emulation option on that drive. There is no requirement that the underlying storage actually be backed by SSDs; this feature can be used with physical media of any type.

For above to work the disk(s) should have the Discard flag set.

If you use the Guest Agent (which you really should) I'd also recommend enabling this under Options > QEMU Guest Agent.

Preventing a full storage

When using thin allocation it can be problematic when a storage reaches 100%. For ZFS you may also want to stay below a certain threshold.
If your storage is already full see this forum post specific to ZFS.

I use a modified version of this snippet to send me a mail if any of my storages reach 75% usage.

# Storage running out of storage. Percentage escaped due to crontab
*/15 * * * * pvesm status 2>&1 | grep -Ev "disabled|error" | tr -d '\%' | awk '$7 >=75 {print $1,$2,$7}' | column -t

Or to check a specific type of storage. LVM-Thin in this case

*/15 * * * * pvesm status 2>&1 | grep "lvmthin" | grep -Ev "disabled|error" | tr -d '\%' | awk '$7 >=75 {print $1,$2,$7}' | column -t

A similar method can be used to check the file system directly for, in this example, at least 100G~ of free space.

*/15 * * * * df /mnt/backupdirectory | tail -n1 | awk '$4 <=100000000 {print $1,$4,$5}' | column -t

Note about paths

It's generally advised to use the full path to executables in cronjobs (like /usr/sbin/pvesm) as PATH is different.
I use this at the top of mine so I don't have to care about that.

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

Useful installer shortcuts/tips

Inside the PVE/PBS installer you can use the following shortcuts.
The terminal is particularily useful in case you need a live environment or do some pre-install customizations.

Shortcut	Info
`CTRL+ALT+F1`	Installer
`CTRL+ALT+F2`	Logs
`CTRL+ALT+F3`	Terminal/Shell
`CTRL+ALT+F4`	Installer GUI

If you press E (see below) you can add args that will be persisted into the installed system.

Temporary kernel arguments

When pressing E during boot/install when the OS/kernel selection shows up you can temporarily edit the kernel arguments. This is useful to debug things or disable passthrough if you run into an issue.
Also see here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#nomodeset_kernel_param

Argument	Info
`nomodeset`	Helps with hangs during boot/install. Nvidia often needs that
`debug`	Debugging messages
`fsck.mode=force`	Triggers a file system check
`systemd.mask=pve-guests.service`	Prevents guests from starting up

Examples of how that selection can look like for PVE with grub/systemd-boot and PBS installer

Passthrough recovery

When passing through devices it can sometimes happen that your device shares an IOMMU group with something else that's important.
It's also possible that groups shift if you exchange a device. All of this can cause a system to become unbootable.
If editing the boot arguments doesn't help, the simplest fix is to go into the UEFI/BIOS and disable every virtualization related thing. VT-x/VT-d/SVM/ACS/IOMMU or whatever it's called for you.

Passthrough tips

For checking IOMMU groups I like this script: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid.
For your convenience

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

To check the hostpci settings of existing VMs (to find which ones use passthrough) you can do this

grep -sR "hostpci" /etc/pve

IOMMU groups PVE CLI

Simple one liner

lspci -vv | grep -P "\d:\d.*|IOMMU"

If you want to use PVE tooling to check the IOMMU groups you can use this

pvesh get /nodes/$(hostname)/hardware/pci --pci-class-blacklist ""

IOMMU groups PVE GUI

To check the IOMMU groups in the GUI you can use the Hardware tab of the VM when adding a PCI(e) device.

Or you can check in Datacenter > Resource Mappings which I think is easier to read because of its tree structure.
It also warns about IOMMU groups.

Rescanning disks/volumes

pct rescan and qm rescan can be useful to find missing volumes and add them to their respective VM/CT.
You can find them as unused disks in Hardware/Resources.

Making KSM start sooner

KSM and ballooning both start when the host reaches 80% memory usage by default.
Ballooning was hardcoded before version 8.4 but it is now configurable via node > System > Options > RAM usage target for ballooning.
To make KSM start sooner and give it a chance to "free" some memory before ballooning starts you can modify /etc/ksmtuned.conf.
For example to let it start at 70% you can configure it like this

KSM_THRES_COEF=30

You can also make it more "aggressive" with something like this

KSM_NPAGES_MAX=5000

Also see official docs:

Enabling a VM's serial console

This allows you to use xterm.js (used for CTs by default) which allows copy & pasting. Tested for debian/ubuntu.
All commands are to be run inside the VM and this might also work for other OSs. Please let me know if it does.

Step 1

Go to the Hardware tab of your VM and add a Serial Port.

Step 2

Some distributions are already set up for this or can be configured via their own UI and this step can be skipped for them.
For example Home Assistant's HAOS is already set up for this and TrueNAS can be configured for it via UI.

Either one of these commands can help finding the right tty.

dmesg -T | grep "tty"
journalctl -b0 -kg "tty"

For example it's ttyS0 for me

Aug 18 02:17:16 nodename kernel: 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A

To enable the TTY edit /etc/default/grub via

nano /etc/default/grub

Find the line starting with GRUB_CMDLINE_LINUX_DEFAULT and add console=ttyS0 console=tty0 at the end (replace ttyS0 with yours from above).

It can look like this for example

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0 console=tty0"

Save via CTRL+X and exit. Afterwards run

update-grub

See here for more:

Step 3

Reboot the VM via the PVE button or power it off and on again to apply the Hardware and bootloader config change.
This is so the VM is cold booted. A normal reboot command from within the VM will not do the same.
You can see if a Hardware change was applied by the color. If it's orange it's still to be applied.

Once that's done your VM should have a functioning xterm.js button under the Console one. Click the arrow beside it.

Importing a OVA/OVF file

This assumes you use the local storage. Replace with whatever Directory storage you want to use.

Go to Datacenter > Storage and modify local to have the Import content type.
Go to local > Import and use the buttons at the top to upload/download a OVA.
Select your OVA/OVF and click the Import button at the top.

I recommend to change the following settings. At least for linux guests.

Advanced > Disks > SCSI Controller > VirtIO SCSI single.
Advanced > Network Interfaces > Model > VirtIO (paravirtualized)

Networking

Prevent NIC name changes

A NIC's (Network Interface Controller/Card) name is hardware dependent and can change when you add or remove PCI(e) devices. Sometimes major kernel upgrades can also cause this.
Since the /etc/network/interfaces file which handles networking uses these names to configure your network, changes to the name will break it.
To prevent those changes you can use a systemd .link file to permanently override the name.
PVE 9 comes with the pve-network-interface-pinning pinning tool.

Network testing

You can show your NICs and assigned ips with ip a/ip l.
If you have a PCI(e) NIC you can use this to show the device(s) and their modules/drivers.

lspci -vnnk | awk '/Ethernet/{print $0}' RS= | grep -Pi --color "^|(?<=Kernel driver in use: |Kernel modules: )[^ ]+"

If you have a USB NIC you can use

lsusb -vt | grep -Pi --color "^|(?<=Driver=)[^,]+"

Just skip the | grep ... with the poor regexes if you don't need to color the output.

This shows the driver used for each NIC. This is useful because it shows the actual name like eno1.

# ls -l /sys/class/net/*/device/driver
lrwxrwxrwx 1 root root 0 May 15 12:58 /sys/class/net/enp6s0/device/driver -> ../../../../../../bus/pci/drivers/igb
lrwxrwxrwx 1 root root 0 May 15 12:58 /sys/class/net/enp7s0/device/driver -> ../../../../../../bus/pci/drivers/igb
lrwxrwxrwx 1 root root 0 May 15 12:58 /sys/class/net/enx00e04c680085/device/driver -> ../../../../../../../bus/usb/drivers/r8152

This shows the device path it belongs to.
Note the values before and after the ->. In this example 06:00.0 and and 07:00.0. enx00e889680195 is a USB device.
You can then cross-reference them with the first column of the lspci | grep -i "Ethernet" or lsusb -vt output

# ls -l /sys/class/net/*/device
lrwxrwxrwx 1 root root 0 Jun 24 12:32 /sys/class/net/enp6s0/device -> ../../../0000:06:00.0
lrwxrwxrwx 1 root root 0 Jun 24 12:32 /sys/class/net/enp7s0/device -> ../../../0000:07:00.0
lrwxrwxrwx 1 root root 0 Jun 24 12:32 /sys/class/net/enx00e889680195/device -> ../../../4-1:1.0

Temporary DHCP

To temporarily use DHCP you can use this

# PVE 8 / Debian 12
ifdown vmbr0; dhclient -v

# When done
dhclient -r; ifup vmbr0

# PVE 9 / Debian 13
ifdown vmbr0; dhcpcd -d

# When done
dhcpcd -k; ifup vmbr0

Optionally pass the NIC name as argument to dhclient/dhcpcd to test a specific one.
This is useful to check general router connectivity or what the subnet/gateway is.
It also allows you to check if your DHCP reservation is properly set up.

Find NIC port

To see which port a network cable is plugged into you can unplug it, run dmesg -Tw to follow the kernel logs and then plug it in again.
Use CTRL+C to stop following the kernel log.

The classic to make the LED blink

# NIC from "DHCPREQUEST for x.x.x.x on NIC_NAME_HERE to x.x.x.yx port 67"
ethtool --identify NIC_NAME_HERE

Not really helpful if you have no network though as ethtool is not pre-installed.

Updating ip

There's multiple ways (GUI or CLI) and multiple files to edit.
You need to edit these files

/etc/network/interfaces (node > System > Network in the GUI)
/etc/hosts (node > System > Hosts in the GUI)
/etc/resolv.conf (node > System > DNS in the GUI)
/etc/issue (Just informational but still a good idea to update it)
/etc/pve/corosync.conf (When in a cluster).

I recommend doing grep -sR "old.ip.here" /etc to check if you missed something. Calling pvebanner, restarting the pvebanner service or rebooting should update the /etc/issue as well. Do this last.
To "reload" /etc/network/interfaces and apply the new ip you can do something like ifreload -av or simply reboot.

GPU passthrough

This will likely never be a complete tutorial, just some often shared commands and tips and scripts.
Consult the following sources for instructions and use mapped devices rather than raw ones.

Make sure to to check the IOMMU groups before passing a device. See above.

Check Device and drivers

Make sure you can see the device and that it uses the expected driver. I.e nvidia, amdgpu, i915, etc.

lspci -vnnk | awk '/VGA/{print $0}' RS= | grep -Pi --color "^|(?<=Kernel driver in use: |Kernel modules: )[^ ]+"

If nvidia devices are not available when the system boots you can work around it by adding this to your crontab

@reboot /usr/bin/nvidia-smi > /dev/null

CT

Create and start your CT before continuing.

Check groups

Check the video and render group ids inside the CT (from the node side). This is important later.
The default ones below should work for debian.

First we define some variables

# CT IDs to check the groups for
CTIDS=(5555 2222 55)

for id in ${CTIDS[@]}; do
    echo "# $id"
    pct exec $id getent group video render | awk -F: '{print $1,$3}'
    echo ""
done

Add devices

This procedure simply calls pct set IDOFCTHERE --devX /givenpath for all the given paths and reboots the CT.
It handles the optional gids (for the video and render groups) when given.
Modify it to add more devices and change the gids. Invalid paths and CTs will be skipped so there's no need to remove anything.

First we define some variables

# CT IDs to add the devices to
CTIDS=(5555 2222 55)

Also see Check which PCI(e) device a drm device belongs to.

# Devices to add to the CT(s)
DEVICES=(
  "/dev/dri/renderD128,gid=104"
  "/dev/dri/renderD129,gid=104"
  "/dev/dri/card0,gid=44"
  "/dev/dri/card1,gid=44"
  "/dev/kfd,gid=104"
  "/dev/nvidia0"
  "/dev/nvidiactl"
  "/dev/nvidia-uvm"
  "/invalid"
  "/dev/nvidia-uvm-tools"
)

Verify and show the group and user ids for the devices on the node. The ids/gids should match with the CT side above. If not modify them.
Note: You can run this inside the CT too.

function showDeviceInfo() {
  echo "user userName group groupName device"
  for device in "${DEVICES[@]}"; do
        trimmedDevice=${device%%,*}

        if [ -e "$trimmedDevice" ]; then
          echo "$(stat -c '%u %U %g %G %n' "$trimmedDevice") $device"
        fi
  done
}

showDeviceInfo | column -t

Run the rest of the script

for ct in $(pct list | awk '/^[0-9]/ {print $1}'); do
  if [[ ! "${CTIDS[@]}" =~ "$ct" ]]; then
    continue
  fi

  echo "# $ct"

  index=0
  for device in "${DEVICES[@]}"; do
      trimmedDevice=${device%%,*}

      if [ -e "$trimmedDevice" ]; then
        echo "pct set $ct --dev${index} $device"
        pct set "$ct" --dev${index} "$device"
        ((index++))
      fi
  done

  echo "pct reboot $ct"
  pct reboot "$ct"
done

Install intel drivers/modules

CT

Some of these packages can be needed for the intel drivers/modules/tools to work properly inside a CT. For example with jellyfin/frigate.
This can be a little bit finicky so I stole part of the list from the helper script project.

apt install -y va-driver-all ocl-icd-libopencl1 intel-opencl-icd vainfo intel-gpu-tools nvtop

Validate with vainfo, intel_gpu_top and nvtop.

Install nvidia drivers/modules via .run file

This installation method gives you more control over the version but you have to update yourself.
These commands should work for both the nodes, VMs and CTs as long as they are based on debian/ubuntu.
This assumes you use the root user. These command are to be run on the node/VM/CT. Copy & paste.

Links and release notes

For datacenter (Some links are broken but you can google for the version)

For linux/unix

Download and install the .run file

<TAB> here means pressing the TAB key to auto complete the file name.

CT

wget LINKFROMABOVEHERE
chmod +x NVIDIA*.run
./NVIDIA<TAB> --no-kernel-modules

VM

wget LINKFROMABOVEHERE
apt install -y linux-headers-generic gcc make dkms 
chmod +x NVIDIA*.run
./NVIDIA<TAB> --dkms

Node

wget LINKFROMABOVEHERE
apt install -y pve-headers gcc make dkms
chmod +x NVIDIA*.run
./NVIDIA<TAB> --dkms

Install nvidia drivers/modules via apt

This is my current recommendation for PVE 9 / Debian 13. If you have to use PVE 8 or Debian 12 see older version of this guide.
It's a simpler method as it uses packages straight from the debian repos. They might be a bit older but this should be fine and it makes installation simpler.
This guide is a bit more opinionated. For example it "forces" you to use the DEB822 format and provides no alternative. Please read the comments for additional hints and options.

Most guides use nvidia's .run files but then you have to update the drivers manually. Instead you can use the drivers/libs from the debian apt repository and update them like any other package.
Note that this has the disadvantage that you, at least by default unless you pin versions, have less control over updates and thus might need to reboot more often. For example when the version of the running driver doesnt match the libraries and tools any more.

These instructions are based on the official debian instructions
I modified them for easy copy pasting. These commands should work for nodes, VMs and CTs as long as they are based on debian. If you use ubuntu please use their docs.
This assumes you use the root user. These command are to be run on the node/VM/CT. Copy & paste.

Prerequisites

We need the non-free component. You should be able to run this to add the component to your /etc/apt/sources.list.d/debian.sources file and update the lists

# Rewrites apt *.list files to *.sources in DEB822 format
apt modernize-sources

# Optional to delete the backup files of the modernize tool above
find /etc/apt/sources.list.d/ -type f -name "*.bak" -delete

# Rewrites the "Components:" line to add non-free and non-free-firmware
sed -i 's/^Components: .*/Components: main contrib non-free non-free-firmware/' /etc/apt/sources.list.d/debian.sources

# Updates the lists
apt update

If your node/VM uses Secure Boot (check with mokutil --sb-state) follow this section.
Make sure to monitor the next boot process via noVNC. You will be asked for the password when importing the key.

apt install dkms && dkms generate_mok

apt install linux-headers-generic

# Set a simple password (a-z keys)
mokutil --import /var/lib/dkms/mok.pub

# If you followed this section after you already installed the driver run this and reboot
# dpkg-reconfigure nvidia-kernel-dkms

Node / VM

apt install nvidia-detect

# Will likely recommend "nvidia-driver"
nvidia-detect

# "nvidia-smi" and "nvtop" are optional but recommended
apt install nvidia-driver nvidia-smi nvtop

CT

Same as above but nvidia-driver is replaced with nvidia-driver-libs.

# "nvidia-smi" and "nvtop" are optional but recommended
apt install nvidia-driver-libs nvidia-smi nvtop

Verify installation

Now see if nvidia-smi works. A reboot might be necessary for the node or a VM.

Post install

Enable Persistence Daemon

This can help save power and decrease access delays. See docs.
These command are to be run on the node or VM. Copy & paste.
Enable and start it with

systemctl enable --now nvidia-persistenced.service

You can see the status in nvidia-smi.

Install and configure Docker Container Toolkit

These command are to be run inside a CT. Copy & paste.
In case you run docker containers inside a CT. Adapted from the official guide.

apt update && apt install -y gpg curl --no-install-recommends
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor > /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

echo 'deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt update
apt install -y nvidia-container-toolkit

nvidia-ctk runtime configure --runtime=docker

# This is needed for LXC or you might get an error like
# nvidia-container-cli: mount error: failed to add device rules: unable to find any existing
# device filters attached to the cgroup: # bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation
# not permitted: unknown
if [[ $(systemd-detect-virt) == "lxc" ]]; then
  sed -i 's/#no-cgroups = false/no-cgroups = true/g' /etc/nvidia-container-runtime/config.toml
fi

systemctl restart docker.service

Now test if it worked

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

ZFS tips

Check space usage and ratios

This sorts by compression ratio

zfs list -ospace,logicalused,compression,compressratio -rS compressratio

This sorts by used size

zfs list -ospace,logicalused,compression,compressratio -rS used

Find old ZFS snapshots

If above shows USEDSNAP being very high and you already deleted snapshots or have none it might be from a old/broken migration task.
It might make sense to add a | less at the end if you have lots of snapshots.

zfs list -ospace,logicalused,compression,compressratio,creation -rs creation -t snap

Update ZFS ARC size

Adapted from the official documentation
PVE uses 10% of the host's memory by default but it only configures the system like that if the OS was installed on ZFS.
If you configure a ZFS storage after installation the defaults of 50% will be used which you probably don't want.
This will change soon: https://bugzilla.proxmox.com/show_bug.cgi?id=6285.
PVE 9 / ZFS 2.3.x removes the 50% limit on linux.

Validate

Check the current ARC size with

arc_summary -s arc

# Also helpful
arcstat

# To check hit ratios
arc_summary -s archits

Check the config file (which might not exist) with

cat /etc/modprobe.d/zfs.conf

Adapt config

The code below will try to not replace your file but only update it.

To calculate a percentage of your total memory in G you can use this

PERCENTAGE=10
grep MemTotal /proc/meminfo | awk -v percentage=$PERCENTAGE '{print int(($2 / 1024^2) / 100 * percentage)}'

Set the size in G to adapt to with this

ARC_SIZE_G=32

Then let the code below do the rest

MEMTOTAL_BYTES="$(($(awk '/MemTotal/ {print $2}' /proc/meminfo) * 1024))"

ARC_SIZE_BYTES_MIN="$(( MEMTOTAL_BYTES / 32 ))"
ARC_SIZE_BYTES_MAX=$(( ARC_SIZE_G * 1024*1024*1024 ))

if [ "$ARC_SIZE_BYTES_MAX" -lt "$ARC_SIZE_BYTES_MIN" ]; then
    echo "Error: Given ARC Size of ${ARC_SIZE_BYTES_MAX} is lower than the current default minimum of ${ARC_SIZE_BYTES_MIN}. Please increase it."
    exit 1
elif [ "$ARC_SIZE_BYTES_MAX" -gt "$MEMTOTAL_BYTES" ]; then
    echo "Error: Given ARC Size of ${ARC_SIZE_BYTES_MAX} is greater than the total memory of ${MEMTOTAL_BYTES}. Please decrease it."
    exit 1
fi

echo "$ARC_SIZE_BYTES_MAX" > /sys/module/zfs/parameters/zfs_arc_max

if grep -q "options zfs zfs_arc_max" "/etc/modprobe.d/zfs.conf" 2> /dev/null; then
    sed -ri "s/.*options zfs zfs_arc_max.*/options zfs zfs_arc_max=$ARC_SIZE_BYTES_MAX # ${ARC_SIZE_G}G/gm" /etc/modprobe.d/zfs.conf
else
    echo -e "options zfs zfs_arc_max=$ARC_SIZE_BYTES_MAX # ${ARC_SIZE_G}G" >> /etc/modprobe.d/zfs.conf
fi

Final steps

Check the config and ARC again to see if everything looks alright, then finally update the initramfs. This is needed so the settings are persisted.

# -k all might not be needed and omitting it speeds up the process
update-initramfs -u -k all

There is no reboot necessary.

Misc tips and scripts

Just some miscellaneous small tips and scripts which don't have a good place yet or are better to be linked from above to keep things structured and organized.

Find unused disks/volumes

If goes without saying that you should be careful here. I trust you have backups.

First rescan

qm rescan
pct rescan

Now find unused disks in the configs

# grep -sR "unused" /etc/pve/
/etc/pve/nodes/pve/qemu-server/500.conf:unused0: nvmezfs:vm-500-disk-1

Investigate their source

# pvesm path nvmezfs:vm-500-disk-1
/dev/zvol/nvmezfs/vm-500-disk-1

Show all of their paths

grep -sR "^unused.*: " /etc/pve/ | awk -F': ' '{print $2}' | xargs -I{} pvesm path {}

Then delete if needed

# pvesm free nvmezfs:vm-500-disk-1
Removed volume 'nvmezfs:vm-500-disk-1'

Delete all of them
Extremely dangerous! List them first as shown above

grep -sR "^unused.*: " /etc/pve/ | awk -F': ' '{print $2}' | xargs -I{} pvesm free {}

Restore guest configs

A script that can extract the .conf file out of pmxcfs's config.db.
Only lightly tested and written without a lot of checks so be careful. Make a backup of the file and install sqlite3 with apt install sqlite3.

#!/usr/bin/env bash
# Attempts to restore .conf files from a PMXCFS config.db file.
set -euo pipefail

# Usually at /var/lib/pve-cluster/config.db
# You can do "cd /var/lib/pve-cluster/" and leave CONFIG_FILE as is
CONFIG_FILE="config.db"

# Using these paths can be convenient but dangerous!
# /etc/pve/nodes/$(hostname)/qemu-server/
VM_RESTORE_PATH="vms"

# /etc/pve/nodes/$(hostname)/lxc/
CT_RESTORE_PATH="cts"

[ -d "$VM_RESTORE_PATH" ] || mkdir "$VM_RESTORE_PATH"
[ -d "$CT_RESTORE_PATH" ] || mkdir "$CT_RESTORE_PATH"

GUESTIDS=$(sqlite3 $CONFIG_FILE "select name from tree where name like '%.conf' and name != 'corosync.conf';")

for guest in $GUESTIDS; do
    sqlite3 $CONFIG_FILE "select data from tree where name like '$guest';" >"$guest"

    if grep -q "rootfs" "$guest"; then
        mv "$guest" "$CT_RESTORE_PATH"
        echo "Restored CT config $guest to $VM_RESTORE_PATH/$guest"
    else
        mv "$guest" "$VM_RESTORE_PATH"
        echo "Restored VM config $guest to $CT_RESTORE_PATH/$guest"
    fi
done

Monitor disk SMART information

You can monitor all your disks' SMART info like this. This creates a nice "table" and highlights changes.

Temperature

watch -x -c -d -n1 bash -c 'for i in /dev/{nvme[0-9]n1,sd[a-z]}; do echo -e "\n[$i]"; smartctl -a $i | grep -Ei "Device Model|Model Number|Serial|temperature"; done'

Errors

watch -x -c -d -n1 bash -c 'for i in /dev/{nvme[0-9]n1,sd[a-z]}; do echo -e "\n[$i]"; smartctl -a $i | grep -Ei "Device Model|Model Number|Serial|error"; done'

Temperature and writes

watch -x -c -d -n1 bash -c 'for i in /dev/{nvme[0-9]n1,sd[a-z]}; do echo -e "\n[$i]"; smartctl -a $i | grep -Ei "Device Model|Model Number|Serial|temperature|writ"; done'

and so on.

Credentials

PVE keeps credentials like CIFS passwords in /etc/pve/priv/storage.

Monitor swap usage

apt install smem --no-install-suggests --no-install-recommends
watch -n1 'smem -atkr -s swap'

Check which PCI(e) device a drm device belongs to

If you have multiple GPUs you will likely have multiple /dev/dri/card* and /dev/dri/renderD* devices.
Note the values before and after the ->. In this example 01:00.0, 05:00.0 and 09:00.0

# ls -l /sys/class/drm/*/device | grep -E "/.*0:"
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/card0/device -> ../../../0000:05:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/card1/device -> ../../../0000:09:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/card2/device -> ../../../0000:01:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/renderD128/device -> ../../../0000:09:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/renderD129/device -> ../../../0000:01:00.0

You can then cross-reference them with the first column of lspci | grep -i "VGA"

# lspci | grep -i "VGA"
01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1)
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c8)

Check which PCI(e) device a disk belongs to

This is useful if you want to know to which controller a disk is connected to.
Note the values before and after the ->. In this example 02:00.1 and 08:00.0

# ls -l /dev/disk/by-path/
lrwxrwxrwx 1 root root  9 Jul  1 18:05 pci-0000:02:00.1-ata-2 -> ../../sda
lrwxrwxrwx 1 root root 13 Jul  1 18:05 pci-0000:08:00.0-nvme-1 -> ../../nvme0n1

You can then cross-reference them with the first column of lspci

# lspci
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge

Note that you can't necessarily rely on the name to always refer to the same device.

IO debugging

This section is about how to check what process and disk causes wait (IO Delay), how fast it reads/writes and so on.
Also see these articles:

General

Install the dependencies first.

apt install -y sysstat iotop-c fatrace

IO Delay

IO delay or IO Wait is shown in the PVE Summary and good ol' top can also be used to check the IO wait via its wa in the CPU column.

iotop-c

iotop-c can show per process statistics. For it to properly work (see why below) you should add the delayacct kernel arg and reboot.

Alternatively, use sysctl kernel.task_delayacct to switch the state at runtime.
Note however that only tasks started after enabling it will have delayacct information.

https://docs.kernel.org/accounting/delay-accounting.html#usage

Run this and check the column (select it via arrow keys) you're interested in.

iotop-c -cP

Also try iotop-c -cPa or press a to toggle cumulative mode and let it run for a while.

iostat

iostat can show per device statistics. Run this and check the %util for the disk(s).

iostat -xyzts --compact --human 1

fatrace

fatrace can be used to check file events such as read, write, create and so on. It can help you identify which processes are modifying files and when. Here's an example to listen for file writes

fatrace -f W

ZFS related

Checking ZFS latency stats

watch -cd -n1 "zpool iostat -yl 1 1"

Checking ZFS queue stats

watch -cd -n1 "zpool iostat -yq 1 1"

Checking ZFS request sizes

watch -cd -n1 "zpool iostat -yr 1 1"

Set up no-subscription apt repositories

With PVE 9 / Debian 13 the file suffix can now also be .sources so don't get confused by that.

GUI

Go to node > Updates > Repositories and add the no-subscription repo.

Disable the enterprise repos

At the end it should look like this.

Go to node > Updates > Refresh and see if everything works as expected.

CLI

PVE 8 / Debian 12

Here's an example /etc/apt/sources.list file

deb http://ftp.debian.org/debian bookworm main contrib
deb http://ftp.debian.org/debian bookworm-updates main contrib

# security updates
deb http://security.debian.org/debian-security bookworm-security main contrib

# Proxmox VE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

To keep the default one and add just the proxmox repo in its own file you can do this

echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list

You should disable the default enterprise repos at this point by commenting out the lines

sed -i '/^#/!s/^/#/' /etc/apt/sources.list.d/pve-enterprise.list

Now check with apt update for errors.

PVE 9 / Debian 13

You can find an example /etc/apt/sources.list.d/proxmox.sources file here.

It looks like this

Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

To create this file you can use this command

cat > /etc/apt/sources.list.d/proxmox.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

You might have to download the key it references via

wget https://enterprise.proxmox.com/debian/proxmox-archive-keyring-trixie.gpg -O /usr/share/keyrings/proxmox-archive-keyring.gpg

You should disable the default enterprise repos at this point by commenting out the lines (or appending Enabled: no)

sed -i '/^#/!s/^/#/' /etc/apt/sources.list.d/pve-enterprise.sources