Definitely prevent stubborn devices from being bound by the host driver in PCI passthrough scenario

prevent-host-driver-in-pci-pass.md

Scenario

You're running a KVM-based virtualization. You want to do PCI/PCIe passthrough of some device. You don't want it to attach to the host OS at all.

Your device looks like this:

00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

Problem

Usually the solutions are simple:

a. If you have only one device listing some module in Kernel modules under lspci -v (e.g. nvidiafb) you can add it to /etc/modprobe.d/some-file.conf as blacklist nvidiafb b. If you have multiple and they're normal devices you just add options vfio-pci ids=8086:1c02 to some file in/etc/modprobe.d/ (make sure to use the vendor id found by running lspci -n -s 00:1f.2, and not the PCI location 00:1f.2)

However, these will not work if your device is handled by something loaded very early, like a driver for your second SATA controller. a. You cannot blacklist ahci (like in example here) because you will prevent all controllers from working (=no boot volume) b. You cannot use modprobe.d to set options because vfio-pci loads much too late.

Solution

There are two prerequisites:

1. vfio-pci must be available before rootfs is attached
2. vfio-pci must load before ahci does

1. The simple part:

• add vfio-pci to /etc/initramfs-tools/modules
• update initramfs: update-initramfs -u -k $(uname -r) This will place the module in the initramfs disk (in /etc/conf/modules)

2. The only slightly harder part: An entry in /etc/initramfs-tools/modules will load vfio-pci before the rootfs is mounted However, /etc/conf/modules from ramdisk is loaded after some scripts (see /init in ramdisk) These scripts (scripts/init-top/) load some drivers... and udev... and udev loads ahci

• create /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci, and enter

  #!/bin/sh
  modprobe vfio-pci ids=8086:1c02

• chmod +x /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci
• edit /usr/share/initramfs-tools/scripts/init-top/udev and change PREREQS="" to PREREQS="load_vfio-pci"
• update initramfs: update-initramfs -u -k $(uname -r) note: this will not work if placed in the "standard place" (/etc/initramfs-tools/scripts...) as dependencies are not cross-directory and /usr/share comes first

Verify

Without the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

With the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: vfio-pci
	Kernel modules: ahci