VFIO Setup on Ubuntu 18.04

ubuntu18.04-vfio.md

Ubuntu 18.04 VFIO GPU passthrough with a single GPU (onboard automatically disables itself)

Hardware

https://www.reddit.com/r/JDM_WAAAT/comments/8zgkfj/server_build_nas_killer_v_20_the_terminator_dual/ 2x X5650 from this thread Aside from that build guide, a generic USB3 card and an AMD R9-200

Grub

Modify /etc/default/grub and add vfio-pci.ids=dead:beef video=efifb:off to GRUB_CMDLINE_LINUX_DEFAULT (use the PCI vid/pid of your primary GPU, not dead:beef)

# My actual
GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity intel_iommu=on vfio-pci.ids=1002:67b0 video=efifb:off"

Run update-grub

Modprobe

Create /etc/modprobe.d/vfio.conf with these lines (this ensures that the vfio-pci module gets loaded before any video drivers):

# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67b0,1002:aac8,1106:3483
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep drm pre: vfio-pci
options kvm_amd avic=1

Run update-initramfs -u

Work around Ubuntu 18.04 bug

# cat /etc/default/grub.d/50-curtin-settings.cfg 
# GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity"
# disable grub os prober that might find other OS installs.
GRUB_DISABLE_OS_PROBER=true
GRUB_TERMINAL=console

Create vfio-pci.conf

# cat /etc/modules-load.d/vfio-pci.conf 
vfio-pci

dmesg outputs

$ dmesg | grep -E "DMAR|IOMMU" 
[    0.000000] ACPI: DMAR 0x00000000BFEC1B94 0001D8 (v01 Intel  OEMDMAR  06040000 LOHR 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.000000] DMAR: Host address width 40
[    0.000000] DMAR: DRHD base: 0x000000fe710000 flags: 0x1
[    0.000000] DMAR: dmar0: reg_base_addr fe710000 ver 1:0 cap c90780106f0462 ecap f0207f
[    0.000000] DMAR: RMRR base: 0x000000bfee3000 end: 0x000000bfeeefff
[    0.000000] DMAR: RMRR base: 0x000000bfef7000 end: 0x000000bfef7fff
[    0.000000] DMAR: RMRR base: 0x000000bfef8000 end: 0x000000bfef8fff
[    0.000000] DMAR: RMRR base: 0x000000bfefd000 end: 0x000000bfefdfff
[    0.000000] DMAR: RMRR base: 0x000000bfef9000 end: 0x000000bfef9fff
[    0.000000] DMAR: RMRR base: 0x000000bfefa000 end: 0x000000bfefafff
[    0.000000] DMAR: RMRR base: 0x000000bfefb000 end: 0x000000bfefbfff
[    0.000000] DMAR: RMRR base: 0x000000bfefc000 end: 0x000000bfefcfff
[    0.000000] DMAR: RMRR base: 0x000000bfefe000 end: 0x000000bfefefff
[    0.000000] DMAR: ATSR flags: 0x0
[    0.000000] DMAR-IR: IOAPIC id 6 under DRHD base  0xfe710000 IOMMU 0
[    0.000000] DMAR-IR: IOAPIC id 7 under DRHD base  0xfe710000 IOMMU 0
[    0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
[    1.230189] DMAR: dmar0: Using Queued invalidation
[    1.230272] DMAR: Setting RMRR:
[    1.230586] DMAR: Setting identity map for device 0000:00:1d.7 [0xbfefe000 - 0xbfefefff]
[    1.230922] DMAR: Setting identity map for device 0000:00:1d.3 [0xbfefc000 - 0xbfefcfff]
[    1.231256] DMAR: Setting identity map for device 0000:00:1d.2 [0xbfefb000 - 0xbfefbfff]
[    1.231594] DMAR: Setting identity map for device 0000:00:1d.1 [0xbfefa000 - 0xbfefafff]
[    1.231928] DMAR: Setting identity map for device 0000:00:1d.0 [0xbfef9000 - 0xbfef9fff]
[    1.232291] DMAR: Setting identity map for device 0000:00:1a.7 [0xbfefd000 - 0xbfefdfff]
[    1.232630] DMAR: Setting identity map for device 0000:00:1a.1 [0xbfef8000 - 0xbfef8fff]
[    1.232967] DMAR: Setting identity map for device 0000:00:1a.0 [0xbfef7000 - 0xbfef7fff]
[    1.233053] DMAR: Setting identity map for device 0000:00:1a.0 [0xbfee3000 - 0xbfeeefff]
[    1.233140] DMAR: Setting identity map for device 0000:00:1a.1 [0xbfee3000 - 0xbfeeefff]
[    1.233226] DMAR: Setting identity map for device 0000:00:1a.7 [0xbfee3000 - 0xbfeeefff]
[    1.233313] DMAR: Setting identity map for device 0000:00:1d.0 [0xbfee3000 - 0xbfeeefff]
[    1.233400] DMAR: Setting identity map for device 0000:00:1d.1 [0xbfee3000 - 0xbfeeefff]
[    1.233487] DMAR: Setting identity map for device 0000:00:1d.2 [0xbfee3000 - 0xbfeeefff]
[    1.233570] DMAR: Setting identity map for device 0000:00:1d.3 [0xbfee3000 - 0xbfeeefff]
[    1.233657] DMAR: Setting identity map for device 0000:00:1d.7 [0xbfee3000 - 0xbfeeefff]
[    1.233742] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.234045] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.234166] DMAR: Intel(R) Virtualization Technology for Directed I/O

$ dmesg | grep -i vfio

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.15.0-32-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro maybe-ubiquity intel_iommu=on vfio-pci.ids=1002:67b0 video=efifb:off
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.15.0-32-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro maybe-ubiquity intel_iommu=on vfio-pci.ids=1002:67b0 video=efifb:off
[    1.568276] VFIO - User Level meta-driver version: 0.3
[    1.573886] vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    1.596052] vfio_pci: add [1002:67b0[ffff:ffff]] class 0x000000/00000000
[  102.286962] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[  102.286970] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0

lspci outputs

(I'm passing through a USB3 card as well as GPU)

$ lspci -nnk | grep -iE "(usb 3|amd)"

02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X] [1002:67b0]
        Kernel modules: radeon, amdgpu
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [Radeon R9 290/290X / 390/390X] [1002:aac8]
04:00.0 USB controller [0c03]: VIA Technologies, Inc. VL805 USB 3.0 Host Controller [1106:3483] (rev 01)
        Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller [1106:3483]

Notes

I also flipped the UEFI switch on the side of the GPU, though I don't know if this made any difference.

22 error due to IOMMU being disabled due to the Grub bug

[    1.984256] VFIO - User Level meta-driver version: 0.3
[    1.990754] vfio-pci: probe of 0000:02:00.0 failed with error -22
[    1.990757] vfio_pci: add [1002:67b0[ffff:ffff]] class 0x000000/00000000
[    1.990768] vfio-pci: probe of 0000:02:00.0 failed with error -22
[    1.990775] vfio-pci: probe of 0000:02:00.1 failed with error -22
[    1.990778] vfio_pci: add [1002:aac8[ffff:ffff]] class 0x000000/00000000
[    1.990788] vfio-pci: probe of 0000:02:00.0 failed with error -22
[    1.990793] vfio-pci: probe of 0000:02:00.1 failed with error -22
[    1.990796] vfio_pci: add [1106:3483[ffff:ffff]] class 0x000000/00000000

Sources

I primarily relied on these two https://www.server-world.info/en/note?os=Ubuntu_18.04&p=kvm&f=11 https://www.reddit.com/r/VFIO/comments/616xih/gpu_passthrough_with_msi_b350_tomahawk/

And referenced this, though I don't know if I used anything. https://www.reddit.com/r/VFIO/comments/4owzs2/gpu_passthrough_with_only_one_gpu/

The -22 error I had prior to finding the StackOverflow post that mentioned the Ubuntu 18.04 bug https://www.reddit.com/r/VFIO/comments/98e6bl/vfio_with_a_board_that_doesnt_have_uefi/

Ubuntu bug: https://serverfault.com/questions/633183/how-do-i-enable-kvm-device-passthrough-in-linux https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1569567

Hi tom, i encounter the -22 error too. and wonders

22 error due to IOMMU being disabled due to the Grub bug

is this shown as un-updated grub.cfg?

I checked my /proc/cmdline and dmesg. seems IOMMU is being enabled properly. but still i got vfio-pci: probe failed with error -22.

Looking forward for your reply :)

It's been five years since I posted this. Unfortunately, I don't remember much of this process.

I believe that error was prior to dealing with the bug in Grub.

I would assume in five years that bug has been dealt with, and current Ubuntu is probably at least somewhat different from 18.02.

I hope you can get things working!

@kalashnlkov were you able to resolve the error?

I got similar error -22, and found out it was caused by bios setting.

The iommu needs to be enabled at bios.

So, on my motherboard, I went to "Advanced>AMD CBS>NBIO Common" and set "IOMMU" to "Enable"

Afterwards, it works nicely.