NixOS: PCI Passthrough

Readme.md

PCI Passthrough

Warning: unfinished (but successfull!)

I did PCI passthrough on Archlinux and Debian with the old PCI-stub method (this was pre-4.0 era). And later I did PCI passthrough on the 4.1+ kernels on Arch and Ubuntu (16.10 I think?).

This is my attempt at doing the same on Nixos.

Requirements

• Supported hardware ✅
• Linux kernel 4.1+ ✅
• PCI IDs of the guest GPU: 1002:67b1, 1002:aac8 ✅
• Patience to figure this out ❓

As you may have noticed, I use the Arch Wiki page about PCI-passthrough as a guide

Current status:

I added the intel_iommu=on to my kernel parameters. (see the nixfile below) I then ran a slightly modified version of the "Ensuring that the groups are valid": ensure_iommu_groups_are_valid.sh (below)

Output:

[redacted]
IOMMU Group 1 	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 	01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290/390] [1002:67b1] (rev 80)
IOMMU Group 1 	01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
[redacted]

My GPU is apparently in the same group as the PCI bridge, but that's no problem since it's the only thing in the group. If you do have the same situation, don't forget to look at the Archwiki: Plugging_your_guest_GPU_in_an_unisolated_CPU-based_PCIe_slot

My PCI IDs are: 1002:67b1 (video) and 1002:aac8 (audio)

I added vfio vfio_iommu_type1 vfio_pci vfio_virqfd to my kernelModules and added options vfio-pci ids=1002:67b1,1002:aac8 as extraModprobeConfig (see nixfile)

After doing journalctl -b | grep -i -e iommu -e dmar I knew that IOMMU was enabled correctly

I added hardcoded paths to the current OVMF store in the qemu config, but it works now! I used this to find it:

ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$/

Then I created a VM to test it (see archwiki instructions, did I mention I use the archwiki?), and after install I added the PCI device.

And it's working!! (but not thouroughly tested)

My previous install (with ubuntu as host) used the disk storage backend, but libvirt on NixOS is currently compiled without support.

TODO

• Make OVMF paths more dynamic (perhaps create a package, or change the libvirt package and add an optional dependency?)
• Test patched libvirt version that is compiled with --with-storage-disk option and publish patch if succesful
• Clean up this guide
• Mention archwiki more

ensuring_iommu_groups_are_valid.sh

#!/bin/bash
# This is a slightly modified version of the script found at 
#    https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid
# It adds a tab to make the output prettier and sorts the output per-group
shopt -s nullglob;
for d in /sys/kernel/iommu_groups/*/devices/*;
do
        n="${d#*/iommu_groups/*}";
        n="${n%%/*}";
        printf 'IOMMU Group %s \t' "$n";
        lspci -nns "${d##*/}";
done | sort -h -k 3

pci-passthrough.nix

# put this file in /etc/nixos/
# change the settings tagged with "CHANGE:"
# and add 
#   ./pci-passthrough.nix
# to /etc/nixos/configuration.nix in `imports`

{config, pkgs, ... }:
{  
  # CHANGE: intel_iommu enables iommu for intel CPUs with VT-d
  # use amd_iommu if you have an AMD CPU with AMD-Vi
  boot.kernelParams = [ "intel_iommu=on" ];
    
  # These modules are required for PCI passthrough, and must come before early modesetting stuff
  boot.kernelModules = [ "vfio" "vfio_iommu_type1" "vfio_pci" "vfio_virqfd" ];
  
  # CHANGE: Don't forget to put your own PCI IDs here
  boot.extraModprobeConfig ="options vfio-pci ids=1002:67b1,1002:aac8";
  
  environment.systemPackages = with pkgs; [
    virtmanager
    qemu
    OVMF
  ];
  
  virtualisation.libvirtd.enable = true;
  virtualisation.libvirtd.enableKVM = true;
  
  # CHANGE: add your own user here
  users.groups.libvirtd.members = [ "root" "your username"];
  
  # CHANGE: use 
  #     ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$//'
  # to find your nix store paths
  virtualisation.libvirtd.qemuVerbatimConfig = ''
    nvram = [
      "/nix/store/_SOME_HASH_-OVMF-2014-12-10/FV/OVMF.fd:/nix/store/_SOME_HASH_-OVMF-2014-12-10/FV/OVMF_VARS.fd"
    ]
  '';

}

Thanks for the guide! A couple of quick notes:

{
  virtualisation.libvirtd.qemuVerbatimConfig = ''
    nvram = [ "${pkgs.OVMF}/FV/OVMF.fd:${pkgs.OVMF}/FV/OVMF_VARS.fd" ]
  '';
}

Also, libvirtd.enableKVM is redundant (it's enabled by default, when libvirtd.enable is true).

Why am I getting Could not detect a default hypervisor inside virt-manager?
Config.: https://github.com/concatime/dotfiles/blob/master/nixos/laptop.nix

@concatime, I think I get the same thing. I always just connect manually through File->Add Connection (keeping default options). Even when I select "Autoconnect," I still have to do it manually each time.

@techhazard, take a look at my fork if you get a chance. I refactored this into a configurable module, among other things. Works like a charm. So all I have to add to my configuration.nix is:

  pciPassthrough = {
    enable = true;
    pciIDs = "10de:13c2,10de:0fbb,1095:3114";
    libvirtUsers = [ "alexj" ];
  };