Ubuntu 21.04 VM with GPU acceleration under Hyper-V...?

Hyper-V GPU-PV with Linux guest.md

Ubuntu 21.04 VM with GPU acceleration under Hyper-V...?

Modern versions of Windows support GPU paravirtualization in Hyper-V with normal consumer graphics cards. This is used e.g. for graphics acceleration in Windows Sandbox, as well as WSLg. In some cases, it may be useful to create a normal VM with GPU acceleration using this feature, but this is not officially supported. People already figured out how to do it with Windows guests though, so why not do the same with Linux? It should be easy given that WSLg is open source and reasonably well documented, right?

Well... not quite. I managed to get it to run... but not well.

How to do it?

1. Verify driver support

Run Get-VMHostPartitionableGpu in PowerShell. You should see your graphics card listed, if you get nothing, update your graphics drivers and try again.

2. Create a new VM in Hyper-V Manager.

Make sure to:

• Use Generation 2
• DISABLE dynamic memory (it interferes with vGPU on Windows so it probably won't work on Linux either, I didn't check this yet though)
• DISABLE automatic snapshots (they are not supported with vGPU and will only cause problems)
• DISABLE secure boot (we'll need custom kernel drivers, and I never tried to make this work with secure boot)
• Don't forget to add more CPU cores because the stupid wizard still adds only one vCPU...

3. Add GPU-PV adapter

From PowerShell running as administrator:

Set-VM -VMName <vmname> -GuestControlledCacheTypes $true -LowMemoryMappedIoSpace 1GB -HighMemoryMappedIoSpace 32GB
Add-VMGpuPartitionAdapter -VMName <vmname>

4. Install Ubuntu 21.04 in the VM like you would usually

5. Build the dxgkrnl driver

Until Microsoft upstreams this driver to the mainline Linux kernel, you will have to build it manually. Use the following script I made to get the driver from the WSL2-Linux-Kernel tree, patch it for out-of-tree build and add it to DKMS:

#!/bin/bash -e
BRANCH=linux-msft-wsl-5.10.y

if [ "$EUID" -ne 0 ]; then
    echo "Swithing to root..."
    exec sudo $0 "$@"
fi

apt-get install -y git dkms

git clone -b $BRANCH --depth=1 https://github.com/microsoft/WSL2-Linux-Kernel
cd WSL2-Linux-Kernel
VERSION=$(git rev-parse --short HEAD)

cp -r drivers/hv/dxgkrnl /usr/src/dxgkrnl-$VERSION
mkdir -p /usr/src/dxgkrnl-$VERSION/inc/{uapi/misc,linux}
cp include/uapi/misc/d3dkmthk.h /usr/src/dxgkrnl-$VERSION/inc/uapi/misc/d3dkmthk.h
cp include/linux/hyperv.h /usr/src/dxgkrnl-$VERSION/inc/linux/hyperv_dxgkrnl.h
sed -i 's/\$(CONFIG_DXGKRNL)/m/' /usr/src/dxgkrnl-$VERSION/Makefile
sed -i 's#linux/hyperv.h#linux/hyperv_dxgkrnl.h#' /usr/src/dxgkrnl-$VERSION/dxgmodule.c
echo "EXTRA_CFLAGS=-I\$(PWD)/inc" >> /usr/src/dxgkrnl-$VERSION/Makefile

cat > /usr/src/dxgkrnl-$VERSION/dkms.conf <<EOF
PACKAGE_NAME="dxgkrnl"
PACKAGE_VERSION="$VERSION"
BUILT_MODULE_NAME="dxgkrnl"
DEST_MODULE_LOCATION="/kernel/drivers/hv/dxgkrnl/"
AUTOINSTALL="yes"
EOF

dkms add dxgkrnl/$VERSION
dkms build dxgkrnl/$VERSION
dkms install dxgkrnl/$VERSION

6. Copy GPU drivers from your host system

Now you will also need to copy some files from the host machine: the closed-source D3D12 implementation provided by Microsoft, as well as Linux parts of the graphics driver provided by your GPU vendor. If you ever tried to run GPU-PV with a Windows guest, this part should look familiar. Figuring out how to transfer the files into the VM is left as an exercise to the reader, I'll just assume that your Windows host volume is available at /mnt for simplicity:

mkdir -p /usr/lib/wsl/{lib,drivers}
cp -r /mnt/Windows/system32/lxss/lib/* /usr/lib/wsl/lib/
cp -r /mnt/Windows/system32/DriverStore/FileRepository/nv_dispi.inf_amd64_* /usr/lib/wsl/drivers/   # this may be different for different GPU vendors, refer to tutorials for Windows guests if needed
chmod -R 0555 /usr/lib/wsl

Note: You will need to repeat this step every time you update Windows or your graphics drivers

7. Set up the system to be able to load libraries from /usr/lib/wsl/lib/:

echo "/usr/lib/wsl/lib" > /etc/ld.so.conf.d/ld.wsl.conf
ldconfig  # (if you get 'libcuda.so.1 is not a symbolic link', just ignore it)

8. Workaround a bug in the D3D12 implementation (it assumes that the /usr/lib/wsl/lib/ mount is case-insensitive... just Windows things...)

ln -s /usr/lib/wsl/lib/libd3d12core.so /usr/lib/wsl/lib/libD3D12Core.so

9. Reboot the VM

If you've done everything correctly, glxinfo | grep "OpenGL renderer string" should display D3D12 (Your GPU Name). If it does not, here are some useful commands for debugging:

sudo lspci -v  # should list the vGPU and the dxgkrnl driver
ls -l /dev/dxg  # should exist if the dxgkrnl
/usr/lib/wsl/lib/nvidia-smi  # should be able to not fail :P

The problems

1. The thing is UNSTABLE. Just running glxgears crashes GNOME, spectacularly. I'd recommend switching to a simple window manager like i3 for testing.
2. GPU acceleration doesn't seem to be picked up everywhere, sometimes it falls back to software rendering with llvmpipe for no apparent reason
3. While when it works you can clearly see that the GPU is working from the FPS counter... I didn't figure out a good way to get these frames from the VM at a reasonable rate yet! The Hyper-V virtual display is slow, and even if you get enhanced session to work it's just RDP under the hood which is not really designed for high FPS output either. On Windows, you can simply use something like Parsec to connect to the VM, but all streaming solutions I know of don't work on a Linux host at all.