PCI Passthrough on Fedora 31

PCI Passthrough

Hardware:

Mainboard ASUS PRIME Z390-A Z390
ASUS Dual-GTX1650-O4G
Intel i9 9900
32GB Ram
NEC Corporation uPD720200 USB 3.0 Host Controller
Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (1 TB)
Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (256 GB)

Operating Systems:

Host: Fedora 31
Guest: Windows 10

Bios Settings:

enable VMX
enable VT-d

Host preparation:

install fedora on the 1TB M.2
use custom partitioning and let the half free for data transfer between host and guest
after installation, update system and install virtualization and uefi bios for the VM:
- dnf install @virtualization
- dnf install edk2-ovmf
prepare grub with nano /etc/default/grub and add to GRUB_CMDLINE_LINUX= at the end:
- intel_iommu=on iommu=pt rd.driver.pre=vfio-pci
activate modules:
- echo "vfio" > /etc/modules-load.d/vfio.conf
- echo "vfio-pci" > /etc/modules-load.d/vfio-pci.conf
- echo "vfio_iommu_type1" > /etc/modules-load.d/vfio_iommu_type1.conf
- echo "vfio_virqfd" > /etc/modules-load.d/vfio_virqfd.conf
reconfigure grub:
- dracut -f --kver $(uname -r)
- grub2-mkconfig > /etc/grub2-efi.cfg
- reboot

check iommu groups and get device IDs:

for g in /sys/kernel/iommu_groups/*; do echo "IOMMU Group ${g##*/}:"; for d in $g/devices/*; do echo -e "\t$(lspci -nns ${d##*/})"; done; done

get all IDs within the group where you devices is member, example output:

IOMMU Group 1:
     00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0d)
     01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU117 [GeForce GTX 1650] [10de:1f82] (rev a1)
     01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10fa] (rev a1)

IDs are: 8086:1901, 10de:1f82, 10de:10fa

for passthrough I need the IDs from graphic card, USB Controller, and the small M.2:
- 8086:1901,10de:1f82,10de:10fa,144d:a804,1033:0194
add this IDs to /etc/modprobe.d/vfio.conf:
- options vfio-pci ids=8086:1901,10de:1f82,10de:10fa,144d:a804,1033:0194
- reconfigure kernel and grub again:
- dracut -f --kver $(uname -r)
- grub2-mkconfig > /etc/grub2-efi.cfg
- reboot
check if VFIO driver are loaded:
- dmesg | grep -i vfio
- and also every device with: lspci -nnk -d 10de:1f82

install VM

use virt-manager
add PCI devices to the VM:
- add hardware > pci host device > ...:
  - add graphic card, with audio device
  - add the small M.2
  - add the USB controller
don't create an virtual hard drive, we use the M.2
don't create many CPUs, create 1 socket, with cores and threads
install windows 10

after installation shutdown VM and edit xml virsh edit vmname:

after editing <features></features> should look like:

<features>
    <acpi/>
    <apic/>
    <hyperv>
        <relaxed state='on'/>
        <vapic state='on'/>
        <spinlocks state='on' retries='8191'/>
        <vendor_id state='on' value='10DE'/>
    </hyperv>
    <kvm>
        <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
</features>

boot guest again and install all update

host tuning

change fedora window manager from wayland to x11 in /etc/gdm/custom.conf:

...
[daemon]
WaylandEnable=false
DefaultSession=gnome-xorg.desktop
...

restart host
install and configure barrier
format the free space from the bigger M.2 and share it with samba

JiangXL/pci-passthrough.md