Arch Linux GPU-Passthrough

A quick guide on how to setup a GPU-Passthorugh. Below are some of my Resources

The Arch Wiki is the goto place for additional information and performance tweaks like CPU-Pinning.

Requirements

IGPU or second dedicated GPU for Host system (unless you want to go the hard way and use one GPU for HOST and GUEST)
Linux Kernel 4.1 or later (vfio-pci)
sudo pacman -S qemu bridge-utils

IOMMU

I/O Memory Management Unit, essential tech for KVM. Your Hardware(CPU, MB-Chipset) better supports it ;)

1 Enable IOMMU

edit /etc/default/grub
append intel_iommu=on or amd_iommu=on to GRUB_CMDLINE_LINUX_DEFAULT
should look similar: GRUB_CMDLINE_LINUX_DEFAULT="[...] intel_iommu=on"
make a new grub config: sudo grub-mkconfig -o /boot/grub/grub.cfg
reboot
execute sudo dmesg | grep -e DMAR -e IOMMU
look for DMAR: IOMMU enabled to ensure IOMMU is enabled
take a look at your IOMMU Groups with this script:

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

identify the target GPU and remember its ID's e.g. 1002:67b1 and 1002:aac8

VFIO

Virtual Function I/O "driver" needed to pass GPUs to a KVM

2.1 Enable VFIO Driver for target GPU

ensure vfio-pci is available: modinfo vfio-pci
edit /etc/modprobe.d/vfio.conf
add this line options vfio-pci ids=1002:67b1,1002:aac8

2.2 Ensure that VFIO will be loaded before other drivers

edit /etc/mkinitcpio.conf
append vfio vfio_iommu_type1 vfio_pci vfio_virqfd to MODULES
VFIO modules must precede other drivers! (if you want to use the GPU on host and guest look at 2.2)
should looke simmilar: MODULES="[...] vfio vfio_iommu_type1 vfio_pci vfio_virqfd [nouveau radeon amdgpu ...]"
ensure modconf is included in HOOKS
should look similar: HOOKS="[...] modconf [...]"
regenerate initramfs: mkinitcpio -p linux
reboot (system might be seem bricked now, I needed to enter BIOS and tell it to use IGPU)
first reboot might take longer since the system might need to figure out stuff. You also might need to switch your monitors input source.
execute dmesg | grep -i vfio andOr lspci -nnkd 1002:67b1 && lspci -nnkd 1002:aac8
verify that vfio-pci has loaded properly and that it is now bound to the right devices

2.3 [OPTIONAL] Dynamic rebinding

This is how you could do a GPU-Passthorugh with only one GPU i guess, but I never tested this without the IGPU. In order to dynamicly use your GPU on the HOST or GUEST you need to load your GPU driver first in /etc/mkinitcpio.conf. Use the following script (Functions taken from Reddit user /u/glowtape) to ensure everything works. When I start a VM, the startscript exits my desktopenviroment binds vfio and then reloads my session and starts the vm. There are reports of People switching drivers with XServer running, but sadly I could not replicate this.

#!/bin/bash

# Which device and which related HDMI audio device. They're usually in pairs.
export VGA_DEVICE=0000:01:00.0
export AUDIO_DEVICE=0000:01:00.1
export VGA_DEVICE_ID=1002:67b1
export AUDIO_DEVICE_ID=1002:aac8

vfiobind() {
	DEV="$1"

	# Check if VFIO is already bound, if so, return.
	VFIODRV="$( ls -l /sys/bus/pci/devices/${DEV}/driver | grep vfio )"
	if [ -n "$VFIODRV" ];
	then
		echo VFIO was already bound to this device!
		return 0
	fi

	echo -n Binding VFIO to ${DEV}...

	echo ${DEV} > /sys/bus/pci/devices/${DEV}/driver/unbind
	sleep 0.5

	echo vfio-pci > /sys/bus/pci/devices/${DEV}/driver_override
	echo ${DEV} > /sys/bus/pci/drivers/vfio-pci/bind
	# echo > /sys/bus/pci/devices/${DEV}/driver_override

	sleep 0.5

	echo OK!
}

vfiounbind() {
	DEV="$1"

	echo -n Unbinding VFIO from ${DEV}...

	echo > /sys/bus/pci/devices/${DEV}/driver_override
	#echo ${DEV} > /sys/bus/pci/drivers/vfio-pci/unbind
	echo 1 > /sys/bus/pci/devices/${DEV}/remove
	sleep 0.2

	echo OK!
}

pcirescan() {

	echo -n Rescanning PCI bus...

	su -c "echo 1 > /sys/bus/pci/rescan"
	sleep 0.2

	echo OK!

}

# Xorg shouldn't run.
if [ -n "$( ps -C xinit | grep xinit )" ];
then
	echo Don\'t run this inside Xorg!
	exit 1
fi

lspci -nnkd $VGA_DEVICE_ID && lspci -nnkd $AUDIO_DEVICE_ID
# Bind specified graphics card and audio device to vfio.
echo Binding specified graphics card and audio device to vfio

vfiobind $VGA_DEVICE
vfiobind $AUDIO_DEVICE

lspci -nnkd $VGA_DEVICE_ID && lspci -nnkd $AUDIO_DEVICE_ID

echo Adios vfio, reloading the host drivers for the passedthrough devices...

sleep 0.5

# Don't unbind audio, because it fucks up for whatever reason.
# Leave vfio-pci on it.
vfiounbind $AUDIO_DEVICE
vfiounbind $VGA_DEVICE

pcirescan

lspci -nnkd $VGA_DEVICE_ID && lspci -nnkd $AUDIO_DEVICE_ID

QEMU

3.1 Setup a Network Bridge

sudo brctl addbr bridge_qemu_0
brctl addif bridge_qemu_0 yourNetworkDevice
ip link set up dev bridge_qemu_0
edit /etc/qemu/bridge.conf
paste allow bridge_qemu_0

3.2 Create a VM with dedicated GPU

create a QCOW2 Drive: qemu-img create -f qcow2 qemu_vm.qcow2 30G
start your VM:

#!/bin/bash
# CPU settings.
# Don't expose KVM as the hypervisor via the MSR hypervisor nodes. (usefull for nvidia drivers)
OPTS="-cpu host,kvm=off"
OPTS="$OPTS -smp 4,sockets=1,cores=4,threads=1"
# Enable KVM full virtualization support.
OPTS="$OPTS -enable-kvm"
# Assign memory to the vm.
OPTS="$OPTS -m 12000"
# VFIO GPU ~and GPU sound~ passthrough.
OPTS="$OPTS -device vfio-pci,host=01:00.0,multifunction=on"
#OPTS="$OPTS -device vfio-pci,host=00:01.1" # dont need audio
# Load our created VM image as a harddrive.
OPTS="$OPTS -hda /home/groot/vm/win10/qemu_vm.qcow2"
# Load our OS setup image e.g. ISO file.
#OPTS="$OPTS -cdrom /home/groot/vm/iso/Win10_1703_English_x64.iso"
# Use the following emulated video device ("none" for disabled).
OPTS="$OPTS -vga qxl"
# Redirect QEMU's console 10.
OPTS="$OPTS -monitor stdio"
# Setup networking, need to try virtio at some point
OPTS="$OPTS -net nic -net bridge,br=bridge_qemu_0"

sudo qemu-system-x86_64 $OPTS

Gictorbit/gpu-passthrough.md