Intel NUC10i7FNH/K + Arch Linux: QEMU/KVM Windows 10 guest with CPU pinning + GPU passthrough with Intel GVT-g + Hugepages

!nuc_arch_qemu_gpu_passthrough

INSTRUCTIONS.md

Intel NUC10i7FNH/K + Arch Linux: QEMU/KVM Windows 10 guest with CPU pinning + GPU passthrough with Intel GVT-g + Hugepages

0. Prerequisites

To increase IGD aperture in BIOS, boot in F2, select Advanced -> Video -> change IGD Aperture Size to 2048 MB

1. Create basic VM

Install required packages:

$ sudo pacman -Syy libvirt qemu virt-manager    # base packages
$ sudo pacman -Syy edk2-ovmf                    # for UEFI/OVMF supports
$ sudo pacman -Syy bridge-utils dnsmaq          # for network bridge

$ systemctl enable --now libvirtd.service virtlogd.service
$ sudo virsh net-start default                  # run this if the NAT interface isn't up automatically

Then create the VM inside virt-manager, namely win10. Remember to set:

• Q35 chip set
• UEFI/OVMF firmware (/usr/share/edk2-ovmf/x64/OVMF_CODE.fd)

2. CPU pinning

Edit the domain with virsh edit:

$ sudo -E virsh edit win10

Add a new <cputune /> item after <vcpu />:

  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='9'/>
    <vcpupin vcpu='6' cpuset='10'/>
    <vcpupin vcpu='7' cpuset='11'/>
    <emulatorpin cpuset='0,6'/>
  </cputune>

Edit <cpu /> item with:

  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
  </cpu>

3. GPU passthrough

3.1. Prepararation

Enable kernel modules in /etc/mkinitcpio.conf:

MODULES=(kvmgt vfio vfio-iommu-type1 vfio-mdev)

and rebuild the initramfs:

$ mkinitcpio -p linux       # for Linux stable kernel
$ mkinitcpio -p linux-lts   # for Linux LTS kernel

Enable kernel parameters in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="... intel_iommu=on i915.enable_gvt=1 kvm.ignore_msrs=1"

and rebuild GRUB configs:

$ sudo grub-mkconfig -o /boot/grub/grub.cfg

Reboot and check if mdev_supported_types exists:

$ ls -la /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types

3.2. Create and assign VGPU

(mainly following the official guide)

Create:

$ sudo /bin/sh -c '\
  GVT_DOM="0000:00"
  GVT_PCI="0000:00:02.0"
  GVT_TYPE="i915-GVTg_V5_4"
  GVT_GUID="<unique-guid>"
  echo "$GVT_GUID" > "/sys/devices/pci$GVT_DOM/$GVT_PCI/mdev_supported_types/$GVT_TYPE/create" \
'

Assign with virsh:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  ...
  <devices>
    ...
    <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
      <source>
        <address uuid='<unique-guid>'/>
      </source>
    </hostdev>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.x-igd-opregion=on'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.romfile=/home/ethanify/VMs/vbios_gvt_uefi.rom'/>
  </qemu:commandline>
</domain>

Run the VM, now Windows will recogise 2 display adapters. Download and install Windows binaries for Spice. After successfully installed, the second adapter will display as Intel Graphic UHD

3.3. Run VM with VGPU:

Edit the domain with virsh. Remember to remove all <graphics /> and <video /> items first:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  ...
  <devices>
    ...
    <graphics type='spice'>
      <listen type='none'/>
      <gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
    </graphics>
    <video>
      <model type='none'/>
    </video>
    <hostdev ... display='on'>
      ...
    </hostdev>
  </devices>
  <qemu:commandline>
    ...
    <qemu:env name='MESA_LOADER_DRIVER_OVERRIDE' value='i965'/>
  </qemu:commandline>
</domain>

Now the VM will run with only Intel Graphic UHD adapter!

4. Hugepages

Create a new file /etc/sysctl.d/10-kvm.conf:

vm.nr_hugepages = 0
vm.nr_overcommit_hugepages = 8192

5. Scripts

Use virt-utils.sh script to:

• set-up mandatory services & procedues prior to VM running
• tear-down the previously set-up services & procedues

virt-utils.sh

#!/usr/bin/env bash
set -o errexit
set -o pipefail
set -o nounset

GVT_DOM="0000:00"
GVT_PCI="0000:00:02.0"
GVT_TYPE="i915-GVTg_V5_2"
GVT_GUID="90986ecd-c716-4e35-8707-50834f85de58"

function set_up() {
  echo "$GVT_GUID" > "/sys/devices/pci$GVT_DOM/$GVT_PCI/mdev_supported_types/$GVT_TYPE/create"
}

function tear_down() {
  echo "1" > "/sys/devices/pci$GVT_DOM/$GVT_PCI/$GVT_GUID/remove"
}

function main() {
  if [ $EUID -ne 0 ]; then
    echo "The utils need to be run as root!" 
    exit 1
  fi

  local sub_command=${1-}

  case "$sub_command" in
    set-up) set_up ;;
    tear-down) tear_down ;;
    *) ;;
  esac
}

main "$@"

win10.xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win10</name>
  <uuid>935cc178-0f54-42bd-85ed-3a93e801c615</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='8'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <emulatorpin cpuset='0,6'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-5.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <runtime state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='partial' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/ethanify/VMs/win10.qcow2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:82:3f:b5'/>
      <source network='default'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice'>
      <listen type='none'/>
      <image compression='off'/>
      <jpeg compression='never'/>
      <zlib compression='never'/>
      <playback compression='off'/>
      <streaming mode='off'/>
      <gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='none'/>
    </video>
    <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'>
      <source>
        <address uuid='90986ecd-c716-4e35-8707-50834f85de58'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.x-igd-opregion=on'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.romfile=/home/ethanify/VMs/vbios_gvt_uefi.rom'/>
    <qemu:env name='MESA_LOADER_DRIVER_OVERRIDE' value='i965'/>
  </qemu:commandline>
</domain>