VFIO Guide for GPU Passthrough

VFIO.md

Introduction:

This guide is for achieving PCI-Passthrough with Intel 7700k and AMD RX 580. My host OS is Manjaro KDE edition, and guest is Windows 10.

Hardware:

Device Type	Device
CPU	Intel Core i7-7700K
Motherboard	ASUS Prime Z270P
RAM	Corsair Vengeance (DDR4 3000 MHz)
GPU (Host)	Intel HD Graphics
GPU (Guest)	AMD Radeon RX 580 OC
Network	TP-Link USB Internet

Operating System:

HOST	Guest
Manjaro	Windows 10 Pro

BIOS Setup

VT-x and VT-d must be enabled in BIOS. Defaults graphics must be set to integrated graphics

Softwares

Run sudo pacman -S libvirt virt-manager ovmf qemu and then run

1. sudo systemctl start libvirtd.service
2. sudo systemctl start virtlogd.socket
3. sudo systemctl enable libvirtd.service
4. sudo systemctl enable virtlogd.socket

Grub setup:

Run sudo nano /etc/default/grub, and add intel_iommu=on to the line GRUB_CMDLINE_LINUX_DEFAULT. Then run sudo update-grub. Reboot, and then run sudo dmesg | grep "Virtualization Technology for Directed I/O". The output should be something like [ 0.610400] DMAR: Intel(R) Virtualization Technology for Directed I/O

Verifying IOMMU Groups:

Run the following as a shell script:

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Output would look something like this:

IOMMU Group 0:
        00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:591f] (rev 05)
IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
        01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7)
        01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 10:
        00:1f.0 ISA bridge [0601]: Intel Corporation 200 Series PCH LPC Controller (Z270) [8086:a2c5]
        00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
        00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
        00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
IOMMU Group 11:
        04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 2:
        00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 630 [8086:5912] (rev 04)

Isolating GPU:

Run sudo nano /etc/modprobe.d/vfio.conf and add the following lines to it

softdep amdgpu pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci
options vfio-pci ids=1002:67df,1002:aaf0

In the last line replace ids with your hardware IDs. Now run sudo nano /etc/mkinitcpio.conf, and add vfio_pci vfio vfio_iommu_type1 vfio_virqfd in the Modules= line.

Blacklisting AMD GPU:

Not doing this step will likely result in X failing to start. System will be stuck in black screen. Run sudo nano /etc/X11/xorg.conf.d/10-intel.conf, and the following to it.

Section "Device"
        Identifier "Intel GPU"
        Driver "modesetting"
        BusID  "PCI:0:2:0"
EndSection

BusID can be obtained by running lspci. Output would contain a line similar to 00:02.0 VGA compatible controller: Intel Corporation HD Graphics 630 (rev 04) BusID is the number in the beginning of this line, but should be written in the PCI:0:2:0 format in 10-intel.conffile.

Running mkinitcpio

After this, run sudo mkinitcpio -g /boot/linux-custom.img, and reboot If everything was succesful, display connected to AMD GPU will no longer display anything.

Automating the above steps:

If you want to frequently enable and disable VFIO, for instance to access GPU on linux, this script can be used to automate the process.

Troubleshooting blackscreen:

If the AMD GPU was not blacklisted, Linux will boot into a blackscreen with error Failed to start Simple Desktop Manager.See 'systemctl sddm.service' for details. In this case, press CTRL+ALT+F2 to get a terminal, login, and remove the vfio.conf file created earlier and reboot. Create the vfio.conf file again and blacklist amdgpu.

Setting up Network for Windows:

Run sudo virt-manager In edit, connection details, click + and add a virtual network of type NAT. Select Forward To and select your network card. This may or may not be necessary, but I did not have internet on windows without this.

Setting up a VM

In virt-manager, create a new VM, select Local Install Media (ISO image or CDROM) and click next. Select your windows 10 iso, and click next. Select appropriate amount of RAM and CPU cores to be allocated. You can use a physical disk for windows install or create a virtual disk. For better performance use a physical disk, preferebly SSD. I allocated a 100 GB virtual disk file for windows. Check Customize configuration before install and give VM a name. In the configuration customization screen, add the following hardware:

1. USB host device: select a mouse to pass to windows.
2. USB host device: select a keyboard to pass to windows.
3. Network: Set network source to virtual network created earlier.
4. Add PCI host device, select your GPU
5. Add PCI host device, select your GPU`s HDMI audio
6. Remove "display" (if any), otherwise your VM might output to a small window rather than the actual display
7. Click Apply

Finishing up:

Run the VM, install windows. Go to device manager and install drivers for your GPU and other hardware, and enjoy your VFIO setup.

Extra Configuration:

Hiding Virtual Machine Status

If you want your VM to not know that its being run in a VM, Enable Copy host configuration in CPU settings of your VM in virt-manager. After that Run sudo EDITOR=nano virsh edit win10. Locate the lines:

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

Keep the top and bottom line and replace everything in between with:

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <feature policy='disable' name='hypervisor'/>
  </cpu>

Sharing Keyboard and Mouse

Install barrier on your host, and start a server. Install it in windows and connect it to your server. For first time configuration:

1. Install barrier on Linux host.
2. Run VM, and install barrier in Windows.
3. Disconnect and reconnect your mouse. This will give your mouse`s control to Linux and keyboard will be attached to Windows.
4. Start Server on Linux.
5. Use Windows as client, enter ther server IP address in server IP, and select autoconfig. Then press start.
6. In Linux, open Barrier log from menu, and copy the Windows machine`s hostname (for example 'DESKTOP-4ETNDIF')
7. Click configure server, add a new screen to the grid, according to the actual positioning of your Windows and Linux Monitors (for example windows on left and linux on right).
8. Set the screen name to the one you have copied from logs.
9. Reload server and client and you should be able to use mouse and keyboard in both.

Note: In order to avoid having to disconnect everytime the guest boots up, after the initial configuration, make the Windows side as server and reload, and then make the linux side as client and reload. You would have to force restart Windows from virt-manager but after that your mouse and keyboard will be automatically shared when windows turns on.

Virtual disk permission error:

If your virtual disk resides on your secondary harddisk, you might get an error for lack of write permissions when starting VM. If this happens, simply remove the device from your VM configuration and add it again.

Credits:

1. ulkeshkosh for their GPU passthrough guide for NVIDIA. This guide is inspired from their guide.
2. u/itoffshore and u/Kneize on r/VFIO for help with blackscreen.
3. This answer by username Daniel B. for help with X.
4. This answer by username Michael Hampton for hiding VM.