Skip to content

Instantly share code, notes, and snippets.

@scyto

scyto/passthrough-setup.md

Last active May 14, 2026 00:14
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save scyto/305224b5e651f6d3c318744bfde99974 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save scyto/305224b5e651f6d3c318744bfde99974 to your computer and use it in GitHub Desktop.
Download ZIP
truenas-vm-promox-prep-and-config
Raw
passthrough-setup.md

pve-nas1 — PCIe Device Inventory & Passthrough Configuration

Reference snapshot of the host's PCIe layout after the May 2026 boot-pool migration and topology shuffle. Use this as the source of truth when rebuilding VM passthrough configs or updating vfio-pci bindings.

Device Inventory

Bus order, identifying who runs each device on the host and how that binding is enforced.

BDF Vendor:Device Description Host Driver Bound by vm.conf index
01:00.0 1e60:2864 Hailo-8 AI Processor vfio-pci modprobe ids= passthrough (via args line in 100.conf)
02:00.0 2646:5024 Kingston DC2000B NVMe vfio-pci modprobe ids= hostpci14
03:00.0 2646:5024 Kingston DC2000B NVMe vfio-pci modprobe ids= hostpci15
21:00.0 10de:2bb1 NVIDIA RTX PRO 6000 Blackwell Workstation Edition MFG mode vfio-pci modprobe ids= hostpci2 (x-vga=0)
42:00.0 1022:7901 AMD FCH SATA Controller (AHCI) — MCIO connector A vfio-pci modprobe ids= + udev rule hostpci0
42:00.1 1022:7901 AMD FCH SATA Controller (AHCI) — MCIO connector B vfio-pci modprobe ids= + udev rule hostpci1
83:00.0 1cc1:8201 ADATA XPG SX8200 Pro NVMe vfio-pci modprobe ids= hostpci7
84:00.0 c0a9:5428 Crucial T710 / Micron 5428 NVMe (4 TB) vfio-pci modprobe ids= hostpci8
a1:00.0 8086:2700 Intel Optane SSD 900P vfio-pci modprobe ids= hostpci3
a3:00.0 8086:2700 Intel Optane SSD 900P vfio-pci modprobe ids= hostpci4
a5:00.0 8086:2700 Intel Optane SSD 900P vfio-pci modprobe ids= hostpci5
a7:00.0 8086:2700 Intel Optane SSD 900P vfio-pci modprobe ids= hostpci6
a9:00.0 (Intel) Intel X550 Ethernet (host management NIC) ixgbe default — (host)
a9:00.1 (Intel) Intel X550 Ethernet (host management NIC) ixgbe default — (host)
ab:00.0 (ASPEED) ASPEED BMC graphics ast default — (host BMC console)
c1:00.0 1344:51c0 Micron 7400 PRO NVMe (rpool mirror leg, serial 22034A6FCF21) nvme default — (host boot pool)
c2:00.0 1344:51c0 Micron 7400 PRO NVMe (rpool mirror leg, serial 220334B1D4BC) nvme default — (host boot pool)
c3:00.0 15b3:1015* Mellanox ConnectX-4 Lx (host data NIC, port 0) mlx5_core default — (host)
c3:00.1 15b3:1015* Mellanox ConnectX-4 Lx (host data NIC, port 1) mlx5_core default — (host)
e1:00.0 1bb1:5018 Seagate E18 NVMe vfio-pci modprobe ids= hostpci9
e2:00.0 1bb1:5018 Seagate E18 NVMe vfio-pci modprobe ids= hostpci10
e3:00.0 1bb1:5018 Seagate E18 NVMe vfio-pci modprobe ids= hostpci11
e4:00.0 1bb1:5018 Seagate E18 NVMe vfio-pci modprobe ids= hostpci12
e6:00.0 1022:7901 AMD FCH SATA Controller (phantom — no MCIO wiring) vfio-pci modprobe ids= (incidental) —
e6:00.1 1022:7901 AMD FCH SATA Controller (phantom — no MCIO wiring) vfio-pci modprobe ids= (incidental) —

* Mellanox device ID listed is the typical value for ConnectX-4 Lx (MT27710); verify with lspci -nn -s c3: if needed.

Configuration Files

/etc/modprobe.d/vfio.conf (or wherever the vfio-pci options live)

options vfio-pci ids=2646:5024,8086:2700,1cc1:8201,1bb1:5018,1e60:2864,1022:7901,10de:2bb1,10de:22e8
Entry Catches
2646:5024 both Kingston DC2000Bs (02:00.0, 03:00.0)
8086:2700 all four Intel Optane 900Ps (a1, a3, a5, a7)
1cc1:8201 ADATA SX8200 Pro (83:00.0)
1bb1:5018 all four Seagate E18s (e1–e4)
1e60:2864 Hailo-8 (01:00.0)
1022:7901 all four AMD SATA controllers (42:00.0/1 + e6:00.0/1); e6:* are phantom and harmless
10de:2bb1 NVIDIA RTX PRO 6000 Blackwell GPU (21:00.0)
10de:22e8 Blackwell HDA audio (no-op while card is in compute mode; future-proof if flipped back to display mode)
c0a9:5428 Crucial T710 / Micron 5428 NVMe (4 TB) (c0a9:5428)

Conspicuously absent (intentional):

  • 1344:51c0 (Micron 7400 PRO) — host boot pool, must NOT be on vfio
  • 15b3:1015 (Mellanox) — host NIC, must NOT be on vfio

/etc/udev/rules.d/99-vfio-udev.rules

ACTION=="add", SUBSYSTEM=="pci", KERNELS=="0000:42:00.0", ATTR{driver_override}="vfio-pci"
ACTION=="add", SUBSYSTEM=="pci", KERNELS=="0000:42:00.0", RUN+="/bin/sh -c 'modprobe vfio-pci; echo 0000:42:00.0 > /sys/bus/pci/drivers/vfio-pci/bind'"
ACTION=="add", SUBSYSTEM=="pci", KERNELS=="0000:42:00.1", ATTR{driver_override}="vfio-pci"
ACTION=="add", SUBSYSTEM=="pci", KERNELS=="0000:42:00.1", RUN+="/bin/sh -c 'modprobe vfio-pci; echo 0000:42:00.1 > /sys/bus/pci/drivers/vfio-pci/bind'"

These pin the two AMD SATA controllers (motherboard MCIO connectors) to vfio-pci by BDF. Strictly speaking, this is redundant with 1022:7901 in the modprobe ids= line — but it's defense-in-depth and ensures the SATA controllers get vfio-bound even if something else interferes with the modprobe binding.

Removed in this session: four ea:00.0 / ea:00.1 rules — those were a leftover attempt at pinning the Mellanox NIC. The Mellanox lives on the host (not passed through), and is now at c3:00.0/.1 regardless.

Apply after edits

update-initramfs -u -k all
udevadm control --reload-rules
# Reboot at next opportunity for modprobe changes to take full effect

VM 100 hostpci Mapping (Proxmox UI)

Slot Configuration
hostpci0 0000:42:00.0,pcie=1,rombar=0
hostpci1 0000:42:00.1,pcie=1,rombar=0
hostpci2 0000:21:00.0,pcie=1,x-vga=0
hostpci3 0000:a1:00.0,pcie=1,rombar=0
hostpci4 0000:a3:00.0,pcie=1,rombar=0
hostpci5 0000:a5:00.0,pcie=1,rombar=0
hostpci6 0000:a7:00.0,pcie=1,rombar=0
hostpci7 0000:83:00.0,pcie=1,rombar=0
hostpci8 0000:84:00.0,pcie=1,rombar=0
hostpci9 0000:e1:00.0,pcie=1,rombar=0
hostpci10 0000:e2:00.0,pcie=1,rombar=0
hostpci11 0000:e3:00.0,pcie=1,rombar=0
hostpci12 0000:e4:00.0,pcie=1,rombar=0
hostpci13 0000:02:00.0,pcie=1,rombar=0
hostpci14 0000:03:00.0,pcie=1,rombar=0

Note the Hailo-8, which is attached via a custom args: line in 100.conf rather than a standard hostpciN entry the args also stop an issue with the the two hostpcie 0 and 1 devices (SATA contollers on 42) args: -set device.hostpci0.x-msix-relocation=bar5 -set device.hostpci1.x-msix-relocation=bar5 -device pcie-root-port,id=pcie_hailo,addr=12.0,bus=pcie.0,chassis=20,hotplug=off -device vfio-pci,host=0000:01:00.0,bus=pcie_hailo,addr=0x0

Key Changes Made This Session

  1. Boot pool rebuilt. rpool had been DEGRADED since Aug 9, 2025 (one half of the mirror was UNAVAIL after a partially-completed migration from Kingston DC2000Bs). Installed the second Micron 7400 PRO (serial 220334B1D4BC) at c2:00.0, replicated the partition table, ran proxmox-boot-tool format/init on the new ESP, and zpool replace'd the dead reference. Pool is now ONLINE.
  2. Second ESP registered. Boot redundancy verified — both 11D1-C69A (nvme1n1p2, original) and 86BA-215A (nvme2n1p2, new) are tracked by proxmox-boot-tool and synced with current kernel.
  3. Mellanox shifted BDF. Adding the second 7400 PRO pushed the Mellanox from c2:00.0/.1 to c3:00.0/.1. Since it lives on the host and is bound by default driver (not by BDF or ID), no config change needed.
  4. vfio ids list cleaned up. Removed stale 10de:1e07 (old Turing GPU). Kept 10de:22e8 (Blackwell audio) as harmless future-proofing.
  5. Udev rules cleaned up. Removed stale ea:00.x rules (Mellanox-targeted, never actually correct since Mellanox is host-side).
  6. Confirmed e6: SATA is phantom.* Both e6:00.0/1 are exposed by the chiplet topology but have no MCIO wiring on this board. They're incidentally bound to vfio via 1022:7901; harmless.
image

Open Items

  • Reboot test to confirm new udev/modprobe config and verify boot from either rpool drive independently.
  • Optional cleanup of stale 7.0.2-2-pve kernel reference (proxmox-boot-tool kernel list / remove).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment