Proxmox VE tips

Just some tips I gathered over time. All in one easily reachable place so I can share it wherever I want.
Please note that unless you see a shebang (#!/...) these code blocks are meant to be copy & pasted.
Some of the steps will not work if you run part of them in a script and copy paste other ones.

Proxmox VE tips

Discard

Using trim/discard with thinly allocated disks gives space back to the storage. This saves space, makes backups faster and is needed for thin allocation to work as expected. This is not related to the storage being backed by an SSD. Use it whenever the storage is thin provisioned. For ZFS this still counts even if Thin Provision (see note below) is not enabled.
Check lvs's Data% column and zfs list's USED/REFER. You might find it to go down when triggering trim as explained below.

Also see official docs:

If you use ZFS you might also want to enable Thin Provisioning in Datacenter > Storage for your ZFS storage.
.
This will only affect newly created disks. Here's how to apply the setting for already existing disks.

CT

Containers usually cannot call fstrim themselves. You can trigger a one time immediate trim via pct fstrim IDOFCTHEREon the node.
I use a cronjob calling pct fstrim.

30 0 * * 0 pct list | awk '/^[0-9]/ {print $1}' | while read ct; do pct fstrim ${ct}; done

Alternatively you can select discard (8.3.x+) as mount option so this happens immediately.

You do not need to enable this for pct fstrim to work.
Use the mount option when you want it to be immediate/continuous and the pct fstrim cronjob to trigger it on a schedule like it usually works for VMs. I prefer the latter.

VM

You can trigger a one time immediate trim via fstrim -av from inside a VM.
Most OSs come with a fstrim.timer which, by default, does a weekly fstrim call.
You can check with systemctl status fstrim.timer. If disabled run systemctl enable fstrim.timer.
To edit it to happen more frequently run systemctl edit fstrim.timer.

Some guest operating systems may also require the SSD Emulation flag to be set. If you would like a drive to be presented to the guest as a solid-state drive rather than a rotational hard disk, you can set the SSD emulation option on that drive. There is no requirement that the underlying storage actually be backed by SSDs; this feature can be used with physical media of any type.

For above to work the disk(s) should have the Discard flag set.

If you use the Guest Agent (which you really should) I'd also recommend enabling this under Options > QEMU Guest Agent.

Preventing a full storage

When using thin allocation it can be problematic when a storage reaches 100%. For ZFS you may also want to stay below a certain threshold.
If your storage is already full see this forum post specific to ZFS.

I use a modified version of this snippet to send me a mail if any of my storages reach 75% usage.

# Storage running out of storage. Percentage escaped due to crontab
*/15 * * * * pvesm status 2>&1 | grep -Ev "disabled|error" | tr -d '\%' | awk '$7 >=75 {print $1,$2,$7}' | column -t

Or to check a specific type of storage. LVM-Thin in this case

*/15 * * * * pvesm status 2>&1 | grep "lvmthin" | grep -Ev "disabled|error" | tr -d '\%' | awk '$7 >=75 {print $1,$2,$7}' | column -t

A similar method can be used to check the file system directly for, in this example, at least 100G~ of free space.

*/15 * * * * df /mnt/backupdirectory | tail -n1 | awk '$4 <=100000000 {print $1,$4,$5}' | column -t

Note about paths

It's generally advised to use the full path to executables in cronjobs (like /usr/sbin/pvesm) as PATH is different.
I use this at the top of mine so I don't have to care about that.

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

Useful installer shortcuts/tips

Inside the PVE/PBS installer you can use the following shortcuts.
The terminal is particularily useful in case you need a live environment or do some pre-install customizations.

Shortcut	Info
`CTRL+ALT+F1`	Installer
`CTRL+ALT+F2`	Logs
`CTRL+ALT+F3`	Terminal/Shell
`CTRL+ALT+F4`	Installer GUI

If you press E (see below) you can add args that will be persisted into the installed system.

Temporary kernel arguments

When pressing E during boot/install when the OS/kernel selection shows up you can temporarily edit the kernel arguments. This is useful to debug things or disable passthrough if you run into an issue.
Also see here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#nomodeset_kernel_param

Argument	Info
`nomodeset`	Helps with hangs during boot/install. Nvidia often needs that
`debug`	Debugging messages
`fsck.mode=force`	Triggers a file system check
`systemd.mask=pve-guests.service`	Prevents guests from starting up

Examples of how that selection can look like for PVE with grub/systemd-boot and PBS installer

Passthrough recovery

When passing through devices it can sometimes happen that your device shares an IOMMU group with something else that's important.
It's also possible that groups shift if you exchange a device. All of this can cause a system to become unbootable.
If editing the boot arguments doesn't help, the simplest fix is to go into the UEFI/BIOS and disable every virtualization related thing. VT-x/VT-d/SVM/ACS/IOMMU or whatever it's called for you.

Passthrough tips

For checking IOMMU groups I like this script: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid.
For your convenience

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

To check the hostpci settings of existing VMs (to find which ones use passthrough) you can do this

grep -sR "hostpci" /etc/pve

IOMMU groups PVE CLI

Simple one liner

lspci -vv | grep -P "\d:\d.*|IOMMU"

If you want to use PVE tooling to check the IOMMU groups you can use this

pvesh get /nodes/$(hostname)/hardware/pci --pci-class-blacklist ""

IOMMU groups PVE GUI

To check the IOMMU groups in the GUI you can use the Hardware tab of the VM when adding a PCI(e) device.

Or you can check in Datacenter > Resource Mappings which I think is easier to read because of its tree structure.
It also warns about IOMMU groups.

Rescanning disks/volumes

pct rescan and qm rescan can be useful to find missing volumes and add them to their respective VM/CT.
You can find them as unused disks in Hardware/Resources.

Making KSM start sooner

KSM and ballooning both start when the host reaches 80% memory usage by default.
Ballooning was hardcoded before version 8.4 but it is now configurable via node > System > Options > RAM usage target for ballooning.
To make KSM start sooner and give it a chance to "free" some memory before ballooning starts you can modify /etc/ksmtuned.conf.
For example to let it start at 70% you can configure it like this

KSM_THRES_COEF=30

You can also make it more "aggressive" with something like this

KSM_NPAGES_MAX=5000

Also see official docs:

Enabling a VM's serial console

This allows you to use xterm.js (used for CTs by default) which allows copy & pasting. Tested for debian/ubuntu.
All commands are to be run inside the VM and this might also work for other OSs. Please let me know if it does.

Step 1

Go to the Hardware tab of your VM and add a Serial Port.

Step 2

Some distributions are already set up for this or can be configured via their own UI and this step can be skipped for them.
For example Home Assistant's HAOS is already set up for this and TrueNAS can be confiured for it via UI.

Follow one one of the methods below. Either one of these commands can help finding the right tty.

dmesg -T | grep "tty"
journalctl -kg "tty"

Method 1

See here for more.

systemctl enable --now serial-getty@ttyS0

Method 2

Run nano /etc/default/grub and add console=ttyS0 console=tty0 to GRUB_CMDLINE_LINUX_DEFAULT.
It can look like this for example

GRUB_CMDLINE_LINUX_DEFAULT="quiet console=ttyS0 console=tty0"

Afterwards run update-grub.

If you use systemd-boot edit /etc/kernel/cmdline instead.

Step 3

Note that a reboot is only needed if you followed method 2 above or the VM was running before you added the serial port.

Reboot the VM via the PVE button or power it off and on again.
This is so the VM is cold booted. A normal reboot command from within the VM will not do the same.
You can see if a Hardware change was applied by the color. If it's orange it's still to be applied.

Once that's done your VM should have a functioning xterm.js button under the Console one. Click the arrow beside it.

Importing a OVA/OVF file

This assumes you use the local storage. Replace with whatever Directory storage you want to use.

Go to Datacenter > Storage and modify local to have the Import content type.
Go to local > Import and use the buttons at the top to upload/download a OVA.
Select your OVA/OVF and click the Import button at the top.

I recommend to change the following settings. At least for linux guests.

Advanced > Disks > SCSI Controller > VirtIO SCSI single.
Advanced > Network Interfaces > Model > VirtIO (paravirtualized)

Networking

Prevent NIC name changes

A NIC's (Network Interface Controller/Card) name is hardware dependent and can change when you add or remove PCI(e) devices. Sometimes major kernel upgrades can also cause this.
Since the /etc/network/interfaces file which handles networking uses these names to configure your network, changes to the name will break it.
To prevent those changes you can use a systemd .link file to permanently override the name.

Network testing

You can show your NICs and assigned ips with ip a/ip l.
If you have a PCI(e) NIC you can use this to show the device(s) and their modules/drivers.

lspci -vnnk | awk '/Ethernet/{print $0}' RS= | grep -Pi --color "^|(?<=Kernel driver in use: |Kernel modules: )[^ ]+"

If you have a USB NIC you can use

lsusb -vt | grep -Pi --color "^|(?<=Driver=)[^,]+"

Just skip the | grep ... with the poor regexes if you don't need to color the output.

This shows the driver used for each NIC. This is useful because it shows the actual name like eno1.

# ls -l /sys/class/net/*/device/driver
lrwxrwxrwx 1 root root 0 May 15 12:58 /sys/class/net/enp6s0/device/driver -> ../../../../../../bus/pci/drivers/igb
lrwxrwxrwx 1 root root 0 May 15 12:58 /sys/class/net/enp7s0/device/driver -> ../../../../../../bus/pci/drivers/igb
lrwxrwxrwx 1 root root 0 May 15 12:58 /sys/class/net/enx00e04c680085/device/driver -> ../../../../../../../bus/usb/drivers/r8152

This shows the device path it belongs to. Cross-reference it with the lspci output.

# ls -l /sys/class/net/*/device
lrwxrwxrwx 1 root root 0 Jun 24 12:32 /sys/class/net/enp6s0/device -> ../../../0000:06:00.0
lrwxrwxrwx 1 root root 0 Jun 24 12:32 /sys/class/net/enp7s0/device -> ../../../0000:07:00.0
lrwxrwxrwx 1 root root 0 Jun 24 12:32 /sys/class/net/enx00e889680195/device -> ../../../4-1:1.0

To temporarily use DHCP you can use this

ifdown vmbr0
dhclient -v

# When done
dhclient -r
ifup vmbr0

Optionally pass the NIC name as argument to dhclient to test a specific one.
This is useful to check general router connectivity or what the subnet/gateway is.
It also allows you to check if your DHCP reservation is properly set up.

To see which port a network cable is plugged into you can unplug it, run dmesg -Tw to follow the kernel logs and then plug it in again.
Use CTRL+C to stop following the kernel log.

The classic to make the LED blink

# NIC from "DHCPREQUEST for x.x.x.x on NIC_NAME_HERE to x.x.x.yx port 67"
ethtool --identify NIC_NAME_HERE

Not really helpful if you have no network though as ethtool is not pre-installed.

Updating ip

There's multiple ways (GUI or CLI) and multiple files to edit.
You need to edit these files

/etc/network/interfaces (node > System > Network in the GUI)
/etc/hosts (node > System > Hosts in the GUI)
/etc/resolv.conf (node > System > DNS in the GUI)
/etc/issue (Just informational but still a good idea to update it)
/etc/pve/corosync.conf (When in a cluster).

I recommend doing grep -sR "old.ip.here" /etc to check if you missed something. Calling pvebanner, restarting the pvebanner service or rebooting should update the /etc/issue as well. Do this last.
To "reload" /etc/network/interfaces and apply the new ip you can do something like ifreload -av or simply reboot.

GPU passthrough

This will likely never be a complete tutorial, just some often shared commands and tips and scripts.
Consult the following sources for instructions and use mapped devices rather than raw ones.

Make sure to to check the IOMMU groups before passing a device. See above.

Check Device and drivers

Make sure you can see the device and that it uses the expected driver. I.e nvidia, amdgpu, i915, etc.

lspci -vnnk | awk '/VGA/{print $0}' RS= | grep -Pi --color "^|(?<=Kernel driver in use: |Kernel modules: )[^ ]+"

If nvidia devices are not available when the system boots you can work around it by adding this to your crontab

@reboot /usr/bin/nvidia-smi > /dev/null

CT

Create and start your CT before continuing.

Check groups

Check the video and render group ids inside the CT (from the node side). This is important later.
The default ones below should work for debian.

First we define some variables

# CT IDs to check the groups for
CTIDS=(5555 2222 55)

for id in ${CTIDS[@]}; do
    echo "# $id"
    pct exec $id getent group video render | awk -F: '{print $1,$3}'
    echo ""
done

Add devices

This procedure simply calls pct set IDOFCTHERE --devX /givenpath for all the given paths and reboots the CT.
It handles the optional gids (for the video and render groups) when given.
Modify it to add more devices and change the gids. Invalid paths and CTs will be skipped so there's no need to remove anything.

First we define some variables

# CT IDs to add the devices to
CTIDS=(5555 2222 55)

Also see Check which PCI(e) device a drm device belongs to.

# Devices to add to the CT(s)
DEVICES=(
  "/dev/dri/renderD128,gid=104"
  "/dev/dri/renderD129,gid=104"
  "/dev/dri/card0,gid=44"
  "/dev/dri/card1,gid=44"
  "/dev/kfd,gid=104"
  "/dev/nvidia0"
  "/dev/nvidiactl"
  "/dev/nvidia-uvm"
  "/invalid"
  "/dev/nvidia-uvm-tools"
)

Verify and show the group and user ids for the devices on the node. The ids/gids should match with the CT side above. If not modify them.
Note: You can run this inside the CT too.

function showDeviceInfo() {
  echo "user userName group groupName device"
  for device in "${DEVICES[@]}"; do
        trimmedDevice=${device%%,*}

        if [ -e "$trimmedDevice" ]; then
          echo "$(stat -c '%u %U %g %G %n' "$trimmedDevice") $device"
        fi
  done
}

showDeviceInfo | column -t

Run the rest of the script

for ct in $(pct list | awk '/^[0-9]/ {print $1}'); do
  if [[ ! "${CTIDS[@]}" =~ "$ct" ]]; then
    continue
  fi

  echo "# $ct"

  index=0
  for device in "${DEVICES[@]}"; do
      trimmedDevice=${device%%,*}

      if [ -e "$trimmedDevice" ]; then
        echo "pct set $ct --dev${index} $device"
        pct set "$ct" --dev${index} "$device"
        ((index++))
      fi
  done

  echo "pct reboot $ct"
  pct reboot "$ct"
done

Install intel drivers/modules

CT

Some of these packages can be needed for the intel drivers/modules/tools to work properly inside a CT. For example with jellyfin/frigate.
This can be a little bit finicky so I stole part of the list from the helper script project.

apt install -y va-driver-all ocl-icd-libopencl1 intel-opencl-icd vainfo intel-gpu-tools nvtop

Validate with vainfo, intel_gpu_top and nvtop.

Install nvidia drivers/modules via .run file

This installation method gives you more control over the version but you have to update yourself.
These commands should work for both the nodes, VMs and CTs as long as they are based on debian/ubuntu.
This assumes you use the root user. These command are to be run on the node/VM/CT. Copy & paste.

Links and release notes

For datacenter (Some links are broken but you can google for the version)

For linux/unix

Download and install the .run file

<TAB> here means pressing the TAB key to auto complete the file name.

CT

wget LINKFROMABOVEHERE
chmod +x NVIDIA*.run
./NVIDIA<TAB> --no-kernel-modules

VM

wget LINKFROMABOVEHERE
apt install -y linux-headers-generic gcc make dkms 
chmod +x NVIDIA*.run
./NVIDIA<TAB> --dkms

Node

wget LINKFROMABOVEHERE
apt install -y pve-headers gcc make dkms
chmod +x NVIDIA*.run
./NVIDIA<TAB> --dkms

Install nvidia drivers/modules via apt

Most guides use nvidia's .run files but then you have to update the drivers manually. Instead you can use nvidia's apt repository and update the drivers like any other package.
Note this has the disadvantage that you, at least by default unless you pin versions, have less control over updates and thus might need to reboot more often.
These instructions are based on the official nvidia instructions
I modified it for easy copy pasting since nvidia's instructions use variables you have to define yourself.
These commands should work for both the nodes, VMs and CTs as long as they are based on debian/ubuntu.
This assumes you use the root user. These command are to be run on the node/VM/CT. Copy & paste.

Prerequisites

We need to make sure the apt lists are up to date and set the variables mentioned above.

apt update

# You'll have to see if you need the additional flags or not
# Todo: Check if this is really needed
apt install -y curl software-properties-common --no-install-recommends --no-install-suggests
add-apt-repository -ny contrib
# Removes the "contrib" component for the proxmox repo as it does not provide it. There has to be a better way
sed -i '/proxmox/s/ contrib//g' /etc/apt/sources.list

source /etc/os-release
# The fancy syntax removes the . from ubuntu's version number
distro=${ID}${VERSION_ID//./}
arch=$(uname -m)

# The package names are different depending on the OS and we only want to install kernel stuff on the node or on VMs
VIRTTYPE=$(systemd-detect-virt)
if [[ "$ID" == "debian" ]]; then
        if [[ $VIRTTYPE == "none" ]]; then
                packages="nvidia-driver-cuda nvidia-kernel-dkms pve-headers"
        elif [[ $VIRTTYPE == "kvm" ]]; then
                packages="nvidia-driver-cuda nvidia-kernel-dkms linux-headers-generic"
        elif [[ $VIRTTYPE == "lxc" ]]; then
                packages="nvidia-driver-cuda"
        fi
elif [[ "$ID" == "ubuntu" ]]; then
        if [[ $VIRTTYPE == "kvm" ]]; then
                packages="gcc cuda-drivers linux-headers-generic"
        elif [[ $VIRTTYPE == "lxc" ]]; then
                packages="cuda-drivers"
        fi
fi

Install the cuda-keyring package

wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb

Enable the apt repository

echo "deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/ /" | tee /etc/apt/sources.list.d/cuda-$distro-$arch.list
apt update

Install driver/modules

# You'll have to see if you need the additional flags or not.
apt -V install $packages --no-install-recommends --no-install-suggests

Install additional useful tools

nvidia-smi is already installed but some people prefer nvtop.

apt install nvtop

Verify installation

Now see if nvidia-smi works. A reboot or manual load of the module(s) via modprobe nvidia might be necessary for the node or a VM.

Post install

Enable Persistence Daemon

This can help save power and decrease access delays. See docs.
These command are to be run on the node or VM. Copy & paste.
Enable and start it with

systemctl enable --now nvidia-persistenced.service

You can see the status in nvidia-smi.

Install and configure Docker Container Toolkit

These command are to be run inside a CT. Copy & paste.
In case you run docker containers inside a CT. Adapted from the official guide.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

echo 'deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt update
apt install -y nvidia-container-toolkit

nvidia-ctk runtime configure --runtime=docker

# This is needed for LXC or you might get an error like
# nvidia-container-cli: mount error: failed to add device rules: unable to find any existing
# device filters attached to the cgroup: # bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation
# not permitted: unknown
if [[ $(systemd-detect-virt) == "lxc" ]]; then
  sed -i 's/#no-cgroups = false/no-cgroups = true/g' /etc/nvidia-container-runtime/config.toml
fi

systemctl restart docker.service

Now test if it worked

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

ZFS tips

Check space usage and ratios

This shows and sorts by compression ratio

zfs list -ospace,logicalused,compression,compressratio -rS compressratio

Update ZFS ARC size

Adapted from the official documentation
PVE uses 10% of the host's memory by default but it only configures the system like that if the OS was installed on ZFS.
If you configure a ZFS storage after installation the defaults of 50% will be used which you probably don't want.
This will change soon: https://bugzilla.proxmox.com/show_bug.cgi?id=6285.

Validate

Check the current ARC size with

arc_summary -s arc

# Also helpful
arcstat

# To check hit ratios
arc_summary -s archits

Check the config file (which might not exist) with

cat /etc/modprobe.d/zfs.conf

Adapt config

The code below will try to not replace your file but only update it.

To calculate a percentage of your total memory in G you can use this

PERCENTAGE=10
grep MemTotal /proc/meminfo | awk -v percentage=$PERCENTAGE '{print int(($2 / 1024^2) / 100 * percentage)}'

Set the size in G to adapt to with this

ARC_SIZE_G=32

Then let the code below do the rest

MEMTOTAL_BYTES="$(($(awk '/MemTotal/ {print $2}' /proc/meminfo) * 1024))"

ARC_SIZE_BYTES_MIN="$(( MEMTOTAL_BYTES / 32 ))"
ARC_SIZE_BYTES_MAX=$(( ARC_SIZE_G * 1024*1024*1024 ))

if [ "$ARC_SIZE_BYTES_MAX" -lt "$ARC_SIZE_BYTES_MIN" ]; then
    echo "Error: Given ARC Size of ${ARC_SIZE_BYTES_MAX} is lower than the current default minimum of ${ARC_SIZE_BYTES_MIN}. Please increase it."
    exit 1
elif [ "$ARC_SIZE_BYTES_MAX" -gt "$MEMTOTAL_BYTES" ]; then
    echo "Error: Given ARC Size of ${ARC_SIZE_BYTES_MAX} is greater than the total memory of ${MEMTOTAL_BYTES}. Please decrease it."
    exit 1
fi

echo "$ARC_SIZE_BYTES_MAX" > /sys/module/zfs/parameters/zfs_arc_max

if grep -q "options zfs zfs_arc_max" "/etc/modprobe.d/zfs.conf" 2> /dev/null; then
    sed -ri "s/.*options zfs zfs_arc_max.*/options zfs zfs_arc_max=$ARC_SIZE_BYTES_MAX # ${ARC_SIZE_G}G/gm" /etc/modprobe.d/zfs.conf
else
    echo -e "options zfs zfs_arc_max=$ARC_SIZE_BYTES_MAX # ${ARC_SIZE_G}G" >> /etc/modprobe.d/zfs.conf
fi

Final steps

Check the config and ARC again to see if everything looks alright, then finally update the initramfs. This is needed so the settings are persisted.

# -k all might not be needed and omitting it speeds up the process
update-initramfs -u -k all

There is no reboot necessary.

Misc tips and scripts

Just some miscellaneous tips and scripts which don't have a good place yet or are better to be linked from above to keep things structured and organized.

Restore guest configs

A script that can extract the .conf file out of pmxcfs's config.db.
Only lightly tested and written without a lot of checks so be careful. Make a backup of the file and install sqlite3 with apt install sqlite3.

#!/usr/bin/env bash
# Attempts to restore .conf files from a PMXCFS config.db file.
set -euo pipefail

# Usually at /var/lib/pve-cluster/config.db
# You can do "cd /var/lib/pve-cluster/" and leave CONFIG_FILE as is
CONFIG_FILE="config.db"

# Using these paths can be convenient but dangerous!
# /etc/pve/nodes/$(hostname)/qemu-server/
VM_RESTORE_PATH="vms"

# /etc/pve/nodes/$(hostname)/lxc/
CT_RESTORE_PATH="cts"

[ -d "$VM_RESTORE_PATH" ] || mkdir "$VM_RESTORE_PATH"
[ -d "$CT_RESTORE_PATH" ] || mkdir "$CT_RESTORE_PATH"

GUESTIDS=$(sqlite3 $CONFIG_FILE "select name from tree where name like '%.conf' and name != 'corosync.conf';")

for guest in $GUESTIDS; do
    sqlite3 $CONFIG_FILE "select data from tree where name like '$guest';" >"$guest"

    if grep -q "rootfs" "$guest"; then
        mv "$guest" "$CT_RESTORE_PATH"
        echo "Restored CT config $guest to $VM_RESTORE_PATH/$guest"
    else
        mv "$guest" "$VM_RESTORE_PATH"
        echo "Restored VM config $guest to $CT_RESTORE_PATH/$guest"
    fi
done

Monitor disk SMART information

You can monitor all your disks' SMART info like this. This creates a nice "table" and highlights changes.

Temperature

watch -x -c -d -n1 bash -c 'for i in /dev/{nvme[0-9]n1,sd[a-z]}; do echo -e "\n[$i]"; smartctl -a $i | grep -Ei "Device Model|Model Number|Serial|temperature"; done'

Errors

watch -x -c -d -n1 bash -c 'for i in /dev/{nvme[0-9]n1,sd[a-z]}; do echo -e "\n[$i]"; smartctl -a $i | grep -Ei "Device Model|Model Number|Serial|error"; done'

Temperature and writes

watch -x -c -d -n1 bash -c 'for i in /dev/{nvme[0-9]n1,sd[a-z]}; do echo -e "\n[$i]"; smartctl -a $i | grep -Ei "Device Model|Model Number|Serial|temperature|writ"; done'

and so on.

Credentials

PVE keeps credentials like CIFS passwords in /etc/pve/priv/storage.

Monitor swap usage

apt install smem --no-install-suggests --no-install-recommends
watch -n1 'smem -atkr -s swap'

Check which PCI(e) device a drm device belongs to

If you have multiple GPUs you will likely have multiple /dev/dri/card* and /dev/dri/renderD* devices.
Here's how to find out from which device they come from. Note the values after the ->.

# ls -l /sys/class/drm/*/device | grep -E "/.*0:"
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/card0/device -> ../../../0000:05:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/card1/device -> ../../../0000:09:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/card2/device -> ../../../0000:01:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/renderD128/device -> ../../../0000:09:00.0
lrwxrwxrwx 1 root root 0 May 17 07:54 /sys/class/drm/renderD129/device -> ../../../0000:01:00.0

Check which PCI(e) device a disk belongs to

This is useful if you want to know to which controller a disk is connected to. Note the values after the ->.
Cross-reference with the output of lspci.

# ls -l /dev/disk/by-path/
lrwxrwxrwx 1 root root  9 Jun 24 12:32 pci-0000:02:00.1-ata-2 -> ../../sda
lrwxrwxrwx 1 root root 13 Jun 24 12:32 pci-0000:08:00.0-nvme-1 -> ../../nvme0n1

You can then check lspci for the matching device

# lspci | grep -i "VGA"
01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1)
05:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c8)

Note that you can't necessarily rely on the name to always refer to the same device.

IO debugging

This section is about how to check what process and disk causes wait (IO Delay), how fast it reads/writes and so on.
Also see these articles:

General

Install the dependencies first.

apt install -y sysstat iotop-c fatrace

IO Delay

IO delay or IO Wait is shown in the PVE Summary and good ol' top can also be used to check the IO wait via its wa in the CPU column.

iotop-c

iotop-c can show per process statistics. For it to properly work you should add the delayacct kernel arg and reboot.
Run this and check the column (select it via arrow keys) you're interested in.

iotop-c -cPo

Also try to press a for cumulative mode and let it run for a while.

iostat

iostat can show per device statistics. Run this and check the %util for the disk(s).

iostat -xyzts --compact --human 1

fatrace

fatrace can be used to check file events such as read, write, create and so on. It can help you identify which processes are modifying files and when. Here's an example to listen for file writes

fatrace -f W

ZFS related

Checking ZFS latency stats

watch -n1 "zpool iostat -yl"

Checking ZFS queue stats

watch -n1 "zpool iostat -yl"

Checking ZFS request sizes

watch -n1 "zpool iostat -yr"

Set up no-subscription apt repositories

GUI

Go to node > Updates > Repositories and add the no-subscription repo.

Disable the enterprise repos

At the end it should look like this.

Go to node > Updates > Refresh and see if everything works as expected.

CLI

You can find a example /etc/apt/sources.list file here. It looks like this

deb http://ftp.debian.org/debian bookworm main contrib
deb http://ftp.debian.org/debian bookworm-updates main contrib

# security updates
deb http://security.debian.org/debian-security bookworm-security main contrib

# Proxmox VE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

To keep the default one and add just the proxmox repo in its own file you can do this

echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list

You should disable the default enterprise repos at this point by commenting out the lines

sed -i '/^#/!s/^/#/' /etc/apt/sources.list.d/{pve-enterprise,ceph}.list

Now check with apt update for errors.

Impact123/Proxmox VE Tips.md