proxmox lxc gpu passthrough

lxc-passthrough.txt

- make sure sriov is enabled in your motherboard bios
- don't use the nvidia gpu as a display (i use asrock rack boards and the crappy on board graphics for local display, which i actually don't use unless there's a problem).
- install nvidia drivers on the host: ./NVIDIA-Linux-x86_64-xxx.xxx.run
- blacklist nvidia and nouvaeu kernel drivers on the host, reboot
- pass the devices through to the LXC before installing drivers, details below (prob doesn't matter, but i do it in this order)
- install THE SAME nvidia drivers on the LXC: ./NVIDIA-Linux-x86_64-xxx.xxx.run --no-kernel-module

below is an example with 4 gpus

find the device info you'll need for passthrough:
> ls -l /dev/fb0
crw-rw---- 1 root video 29, 0 Sep 27 22:01 /dev/fb0
> ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Sep 27 22:01 /dev/nvidia0
crw-rw-rw- 1 root root 195,   1 Sep 27 22:01 /dev/nvidia1
crw-rw-rw- 1 root root 195,   2 Sep 27 22:01 /dev/nvidia2
crw-rw-rw- 1 root root 195,   3 Sep 27 22:01 /dev/nvidia3
crw-rw-rw- 1 root root 195, 255 Sep 27 22:01 /dev/nvidiactl
crw-rw-rw- 1 root root 509,   0 Sep 27 22:01 /dev/nvidia-uvm
crw-rw-rw- 1 root root 509,   1 Sep 27 22:01 /dev/nvidia-uvm-tools
/dev/nvidia-caps:
cr-------- 1 root root 235, 1 Sep 27 22:01 nvidia-cap1
cr--r--r-- 1 root root 235, 2 Sep 27 22:01 nvidia-cap2
ls -l /dev/dri/
drwxr-xr-x 2 root root        220 Sep 27 22:01 by-path
crw-rw---- 1 root video  226,   0 Sep 27 22:01 card0
crw-rw---- 1 root video  226,   1 Sep 27 22:01 card1
crw-rw---- 1 root video  226,   2 Sep 27 22:01 card2
crw-rw---- 1 root video  226,   3 Sep 27 22:01 card3
crw-rw---- 1 root video  226,   4 Sep 27 22:01 card4
crw-rw---- 1 root render 226, 128 Sep 27 22:01 renderD128
crw-rw---- 1 root render 226, 129 Sep 27 22:01 renderD129
crw-rw---- 1 root render 226, 130 Sep 27 22:01 renderD130
crw-rw---- 1 root render 226, 131 Sep 27 22:01 renderD131
note the device major and minor numbers above. you'll need them below.
you may need to manually load the kernel modules the first time, or on restarts:
modprobe nvidia
modprobe nvidia-uvm
modprobe nvidiafb
mknod -m 666 /dev/nvidia-uvm c $(grep nvidia-uvm /proc/devices | awk '{print $1}') 0

typically on restarts you'll need to run nvidia-smi - set up a job to do this on restarts and that should do it for you.
LXC device passthrough. i pass everything related through, you may or may not need everything but i'd rather not pick and choose.
ssh into the host
open the LXC config to edit
vi /etc/pve/lxc/id.conf
at the bottom of the file, add:
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 509:* rwm
lxc.cgroup2.devices.allow: c 235:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card1 dev/dri/card1 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card2 dev/dri/card2 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card3 dev/dri/card3 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card4 dev/dri/card4 none bind,optional,create=file
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD129 dev/renderD129 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD130 dev/renderD130 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD131 dev/renderD131 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia3 dev/nvidia3 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps dev/nvidia-caps none bind,optional,create=dir

after this you should be able to start up your LXC, install drivers and if you run nvidia-smi in it, you'll see all of the GPUs. i think that's everything.