Definitely prevent stubborn devices from being bound by the host driver in PCI passthrough scenario

prevent-host-driver-in-pci-pass.md

Scenario

You're running a KVM-based virtualization. You want to do PCI/PCIe passthrough of some device. You don't want it to attach to the host OS at all.

Your device looks like that:

00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

Problem

Usually the solutions are simple:

1. If you have only one device listing some module in Kernel modules (e.g. nvidiafb) you can add it to /etc/modprobe.d/some-file.conf as blacklist nvidiafb
2. If you have multiple and they're normal devices you just add options vfio-pci ids=8086:1c02 to some file in/etc/modprobe.d/ (make sure to use the id in [...] and not pci location 00:1f.2)

However, these will not work if your device is handled by something loaded very very VERY early... like a driver for your second SATA controller.

1. You cannot blacklist ahci (like in example here) because you will prevent all controllers from working (=no boot volume)
2. You cannot use modprobe.d to set options because vfio-pci loads waaaaay too late.

Solution

There are two prerequisites:

1. vfio-pci must be availbale before rootfs is attached
2. vfio-pci must load before ahci loads

The first is simple:

• add vfio-pci to /etc/initramfs-tools/modules
• update initramfs: update-initramfs -u -k $(uname -r)
• Proxmox on UEFI: if you're using Proxmox 7 booted using UEFI mode you also need to run proxmox-boot-tool refresh
• it will place the module in initramfs disk (in /etc/conf/modules)

The second is more complicated:

• entry in /etc/initramfs-tools/modules will load vfio-pci before the rootfs is mounted
• however, /etc/conf/modules from ramdisk is loaded after some scripts (see /init in ramdisk)
• these scripts (scripts/init-top/) load some drivers... and udev... and udev loads ahci
• solution:
  • create /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci with

    #!/bin/sh
    modprobe vfio-pci ids=8086:1c02

  • chmod +x /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci
  • edit /usr/share/initramfs-tools/scripts/init-top/udev and change PREREQS="" to PREREQS="load_vfio-pci"
• update initramfs: update-initramfs -u -k $(uname -r)
• Proxmox on UEFI: if you're using Proxmox 7 booted using UEFI mode you also need to run proxmox-boot-tool refresh
• note: this will not work if placed in "standard place" (/etc/initramfs-tools/scripts...) as dependencies are not cross-directory and /usr/share comes first

Verify

Without the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

With the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: vfio-pci
	Kernel modules: ahci

Hey there,

I just wanted to leave a very big and kind THANK YOU. I have a virtualized TrueNAS Scale VM on my Proxmox server currently running 8.2.4 which was always trying to import the zpool that belongs to TrueNAS.
I thought it would be easy to prevent Proxmox from seeing the passed through SATA controller by adding it to vfio.conf, but softdep did not work (it kept asking me to import the pool manually at initramfs). Only this gist did the trick.