Modern versions of Windows support GPU paravirtualization in Hyper-V with normal consumer graphics cards. This is used e.g. for graphics acceleration in Windows Sandbox, as well as WSLg. In some cases, it may be useful to create a normal VM with GPU acceleration using this feature, but this is not officially supported. People already figured out how to do it with Windows guests though, so why not do the same with Linux? It should be easy given that WSLg is open source and reasonably well documented, right?
Well... not quite. I managed to get it to run... but not well.
- Verify driver support
Run Get-VMHostPartitionableGpu in PowerShell. You should see your graphics card listed, if you get nothing, update your graphics drivers and try again.
- Create a new VM in Hyper-V Manager.
Make sure to:
- Use Generation 2
- DISABLE dynamic memory (it interferes with vGPU on Windows so it probably won't work on Linux either, I didn't check this yet though)
- DISABLE automatic snapshots (they are not supported with vGPU and will only cause problems)
- DISABLE secure boot (we'll need custom kernel drivers, and I never tried to make this work with secure boot)
- Don't forget to add more CPU cores because the stupid wizard still adds only one vCPU...
- Add GPU-PV adapter
From PowerShell running as administrator:
Set-VM -VMName <vmname> -GuestControlledCacheTypes $true -LowMemoryMappedIoSpace 1GB -HighMemoryMappedIoSpace 32GB
Add-VMGpuPartitionAdapter -VMName <vmname>-
Install Ubuntu 21.04 in the VM like you would usually
-
Build the
dxgkrnldriver
Until Microsoft upstreams this driver to the mainline Linux kernel, you will have to build it manually. Use the following script I made to get the driver from the WSL2-Linux-Kernel tree, patch it for out-of-tree build and add it to DKMS:
#!/bin/bash -e
BRANCH=linux-msft-wsl-5.10.y
if [ "$EUID" -ne 0 ]; then
echo "Swithing to root..."
exec sudo $0 "$@"
fi
apt-get install -y git dkms
git clone -b $BRANCH --depth=1 https://github.com/microsoft/WSL2-Linux-Kernel
cd WSL2-Linux-Kernel
VERSION=$(git rev-parse --short HEAD)
cp -r drivers/hv/dxgkrnl /usr/src/dxgkrnl-$VERSION
mkdir -p /usr/src/dxgkrnl-$VERSION/inc/{uapi/misc,linux}
cp include/uapi/misc/d3dkmthk.h /usr/src/dxgkrnl-$VERSION/inc/uapi/misc/d3dkmthk.h
cp include/linux/hyperv.h /usr/src/dxgkrnl-$VERSION/inc/linux/hyperv_dxgkrnl.h
sed -i 's/\$(CONFIG_DXGKRNL)/m/' /usr/src/dxgkrnl-$VERSION/Makefile
sed -i 's#linux/hyperv.h#linux/hyperv_dxgkrnl.h#' /usr/src/dxgkrnl-$VERSION/dxgmodule.c
echo "EXTRA_CFLAGS=-I\$(PWD)/inc" >> /usr/src/dxgkrnl-$VERSION/Makefile
cat > /usr/src/dxgkrnl-$VERSION/dkms.conf <<EOF
PACKAGE_NAME="dxgkrnl"
PACKAGE_VERSION="$VERSION"
BUILT_MODULE_NAME="dxgkrnl"
DEST_MODULE_LOCATION="/kernel/drivers/hv/dxgkrnl/"
AUTOINSTALL="yes"
EOF
dkms add dxgkrnl/$VERSION
dkms build dxgkrnl/$VERSION
dkms install dxgkrnl/$VERSION- Copy GPU drivers from your host system
Now you will also need to copy some files from the host machine: the closed-source D3D12 implementation provided by Microsoft, as well as Linux parts of the graphics driver provided by your GPU vendor. If you ever tried to run GPU-PV with a Windows guest, this part should look familiar. Figuring out how to transfer the files into the VM is left as an exercise to the reader, I'll just assume that your Windows host volume is available at /mnt for simplicity:
mkdir -p /usr/lib/wsl/{lib,drivers}
cp -r /mnt/Windows/system32/lxss/lib/* /usr/lib/wsl/lib/
cp -r /mnt/Windows/system32/DriverStore/FileRepository/nv_dispi.inf_amd64_* /usr/lib/wsl/drivers/ # this may be different for different GPU vendors, refer to tutorials for Windows guests if needed
chmod -R 0555 /usr/lib/wslNote: You will need to repeat this step every time you update Windows or your graphics drivers
- Set up the system to be able to load libraries from
/usr/lib/wsl/lib/:
echo "/usr/lib/wsl/lib" > /etc/ld.so.conf.d/ld.wsl.conf
ldconfig # (if you get 'libcuda.so.1 is not a symbolic link', just ignore it)- Workaround a bug in the D3D12 implementation (it assumes that the
/usr/lib/wsl/lib/mount is case-insensitive... just Windows things...)
ln -s /usr/lib/wsl/lib/libd3d12core.so /usr/lib/wsl/lib/libD3D12Core.so- Reboot the VM
If you've done everything correctly, glxinfo | grep "OpenGL renderer string" should display D3D12 (Your GPU Name). If it does not, here are some useful commands for debugging:
sudo lspci -v # should list the vGPU and the dxgkrnl driver
ls -l /dev/dxg # should exist if the dxgkrnl
/usr/lib/wsl/lib/nvidia-smi # should be able to not fail :P- The thing is UNSTABLE. Just running
glxgearscrashes GNOME, spectacularly. I'd recommend switching to a simple window manager like i3 for testing. - GPU acceleration doesn't seem to be picked up everywhere, sometimes it falls back to software rendering with llvmpipe for no apparent reason
- While when it works you can clearly see that the GPU is working from the FPS counter... I didn't figure out a good way to get these frames from the VM at a reasonable rate yet! The Hyper-V virtual display is slow, and even if you get enhanced session to work it's just RDP under the hood which is not really designed for high FPS output either. On Windows, you can simply use something like Parsec to connect to the VM, but all streaming solutions I know of don't work on a Linux host at all.
is the source only work in kernel 6.1 ? I tried 6.6 branch and there is code problem . It use wrong GUID compare feature that makes GUID doesn't match device type.
However, after I fix problem ,I found the nvidia-smi still doesn't work :
/usr/lib/wsl/lib/nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Failed to properly shut down NVML: Driver Not Loaded
I am sure the previous steps are correct because I can find /dev/dxg and after I enalbe the driver trace log I can findout these:
[ 5793.684936] dxgk: 000000007d7a5f0b 18484 18484
[ 5793.684943] dxgk: new dxgprocess created
[ 5793.685001] dxgk: send_sync_msg global: 0 0000000033a39403 584 4
[ 5793.685033] dxgk: waiting completion: 2
[ 5793.685138] dxgk: New adapter message: 0000000000000000
[ 5793.685140] dxgk: next packet (id, size, type): 2 8 11
[ 5793.685236] dxgk: completion done: 2 0
[ 5793.685239] dxgk: create_process returned 40000640
[ 5793.685267] dxgk: unlocked ioctl c018473e Code:62
[ 5793.685270] dxgk: found 0 adapters
[ 5793.685271] dxgk: Ioctl returned: 0
[ 5793.685272] dxgk: Ioctl returned: 0
[ 5793.685273] dxgk: unlocked ioctl c018473e Code:62
[ 5793.685279] dxgk: creating new process adapter info
[ 5793.685280] dxgk: 000000002497c92d 0000000001131e0e
[ 5793.685287] dxgk: adapter: 40000000 1:9e1a2fa2
[ 5793.685288] dxgk: found 1 adapters
[ 5793.685290] dxgk: Ioctl returned: 0
[ 5793.685291] dxgk: Ioctl returned: 0
[ 5793.685295] dxgk: unlocked ioctl c0184709 Code:9
[ 5793.685296] dxgk: Type: 13 Size: 4
[ 5793.685298] dxgk: send_sync_msg global: 0 000000003f7af329 59 8
[ 5793.685303] dxgk: waiting completion: 3
[ 5793.685438] dxgk: New adapter message: 0000000000000000
[ 5793.685440] dxgk: next packet (id, size, type): 3 8 11
[ 5793.685495] dxgk: completion done: 3 0
[ 5793.685497] dxgk: Ioctl returned: 0
[ 5793.685501] dxgk: unlocked ioctl c0184709 Code:9
[ 5793.685502] dxgk: Type: 48 Size: 430
[ 5793.685506] dxgk: send_sync_msg global: 0 0000000015916210 1127 1076
[ 5793.685510] dxgk: waiting completion: 4
[ 5793.685645] dxgk: New adapter message: 0000000000000000
[ 5793.685647] dxgk: next packet (id, size, type): 4 1080 11
[ 5793.685701] dxgk: completion done: 4 0
[ 5793.685705] dxgk: Ioctl returned: 0
[ 5793.685910] dxgk: 000000007d7a5f0b, 00000000974f4c77
[ 5793.685913] dxgk: 000000002497c92d 0000000001131e0e
[ 5793.685914] dxgk: 000000000f25234a 40000000
[ 5793.685918] dxgk: send_sync_msg global: 0 00000000fb3ef67d 40 4
[ 5793.685922] dxgk: waiting completion: 5
[ 5793.686019] dxgk: New adapter message: 0000000000000000
[ 5793.686021] dxgk: next packet (id, size, type): 5 8 11
[ 5793.686035] dxgk: completion done: 5 0
package type 11 means complete (according to the source code enum define string).
Then ,I think this log means: I have successfully created /dev/dxg and have used correct nvidia-smi to send package through MS vmbus, most case it should make everything work.
But NO.
My nvidia-smi still doesn't work and my glxinfo and vulkaninfo still doesn't found new renderer.
The only reason I can image is : for some reason, this driver only work on kernel 6.1.
I just wanna show what I found before I swith my kernel to 6.1-- because it will make more work for me.
maybe I will report the result after I checked it on kernel 6.1.