QEMU with VirtIO GPU Vulkan Support

qemu-vulkan-virtio.md

QEMU with VirtIO GPU Vulkan Support

With its latest reales qemu added the Venus patches so that virtio-gpu now support venus encapsulation for vulkan. This is one more piece to the puzzle towards full Vulkan support.

An outdated blog post on clollabora described in 2021 how to enable 3D acceleration of Vulkan applications in QEMU through the Venus experimental Vulkan driver for VirtIO-GPU with a local development environment. Following up on the outdated write up, this is how its done today.

Definitions

Let's start with the brief description of the projects mentioned in the post & extend them:

• QEMU is a machine emulator
• VirGL is an OpenGL driver for VirtIO-GPU, available in Mesa.
• Venus is an experimental Vulkan driver for VirtIO-GPU, also available in Mesa.
• Virglrenderer is a library that enables hardware acceleration to VM guests, effectively translating commands from the two drivers just mentioned to either OpenGL or Vulkan.
• libvirt is an API for managing platform virtualization
• virt-manager is a desktop user interface for managing virtual machines through libvirt

Merged Patches:

• 2024-08-14 venus: make cross-device optional merged in mesa 24.2
• 2024-11-25 KVM: Stop grabbing references to PFNMAP'd pages merged in linux 6.13
• 2024-11-12 Support blob memory and venus on qemu merged in qemu 9.2.0

Work in progress:

• libvirt Add support for more virtio-vga-gl arguments #638
• virt-manager Add support for Venus / Vulkan VirtIO-GPU driver #362

Prerequisites

Make sure you have the proper version installed on the host:

• linux kernel >= 6.13 built with CONFIG_UDMABUF
• working Vulkan and kvm setup
• qemu >= 9.2.0
• virglrenderer with enabled venus support
• mesa >= 24.2.0

You can verify this like so:

$ uname -r
6.13.0
$ ls /dev/udmabuf
/dev/udmabuf
$ ls /dev/kvm
/dev/kvm
$ qemu-system-x86_64 --version
QEMU emulator version 9.2.0
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

Check your distros package sources how they build virglrenderer.

For Vulkan to work you need the proper drivers to be installed for your graphics card.

To verfiy your setup, install vulkan-tools. Make sure mesa >= 24.2.0 and test vkcube:

$ vulkaninfo --summary | grep driverInfo
	driverInfo         = Mesa 24.2.3-1ubuntu1
	driverInfo         = Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0)
...
$ vkcube
Selected GPU x: ..., type: ...

Building qemu

If your distro doesn't (yet) ship and updated version of qemu, you can build it yourself from source:

wget https://download.qemu.org/qemu-9.2.0.tar.xz
tar xvJf qemu-9.2.0.tar.xz
cd qemu-9.2.0
mkdir build && cd build
../configure --target-list=x86_64-softmmu  \
  --enable-kvm                 \
  --enable-opengl              \
  --enable-virglrenderer       \
  --enable-gtk                 \
  --enable-sdl
make -j4

The configuration step will throgh errors if packages are missing. Check the qemu wiki for further info what to install: https://wiki.qemu.org/Hosts/Linux

Create and run an image for QEMU

Create an image & fetch the distro of your choice:

Host

ISO=ubuntu-24.10-desktop-amd64.iso  
wget https://releases.ubuntu.com/oracular/ubuntu-24.10-desktop-amd64.iso  

IMG=ubuntu-24-10.qcow2
qemu-img create -f qcow2 $IMG 16G

Run a live version or install the distro

qemu-system-x86_64                                               \
    -enable-kvm                                                  \
    -M q35                                                       \
    -smp 4                                                       \
    -m 4G                                                        \
    -cpu host                                                    \
    -net nic,model=virtio                                        \
    -net user,hostfwd=tcp::2222-:22                              \
    -device virtio-vga-gl,hostmem=4G,blob=true,venus=true        \
    -vga none                                                    \
    -display gtk,gl=on,show-cursor=on                            \
    -usb -device usb-tablet                                      \
    -object memory-backend-memfd,id=mem1,size=4G                 \
    -machine memory-backend=mem1                                 \
    -hda $IMG                                                    \
    -cdrom $ISO                                                  

Adjust the parameters accordingly:

• smp: number of cpu cores
• m: RAM
• hostmem,size: VRAM

Guest

Install mesa-utilites and vulkan-tools to test the setup:

$ glxinfo -B

$ vkcube
Selected GPU x: ..., type: ...

If the deive is llvmpipe somehting is wrong. The device should be virgl (...).

Troubleshooting

• (host) add -d guest_errors to show error messages from the guest
• (guest) try installing vulkan virtio drivers and mesa
• check the original blog post

Ubuntu 24.10

This is how you do it on Ubuntu

kernel

Install mainline: https://github.com/bkw777/mainline

sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline

find the latest kernel (>= 6.13), at the time of writing 6.13 is a release candidate, so include those:

$ mainline check --include-rc

Install kernel:

$ sudo mainline install 6.13-rc1

Verfify installed kernels:

$ mainline list-installed
mainline 1.4.10
Installed Kernels:
linux-image-6.11.0-13-generic
linux-image-generic-hwe-24.04
linux-image-unsigned-6.13.0-061300rc1-generic
mainline: done

reboot into new kernel

verify running kernel

$ uname -r
6.13.0-061300rc1-generic

virglrenderer

the ubuntu package is not compiled with the proper flags.

If installed remove it: $ sudo apt-get remove libvirglrenderer-dev

download, build & install from source with venus enabled

wget    https://gitlab.freedesktop.org/virgl/virglrenderer/-/archive/1.1.0/virglrenderer-1.1.0.tar.gz
sudo apt-get install python3-full ninja-build libvulkan-dev libva-dev
python3 -m venv venv
venv/bin/pip install meson
venv/bin/meson build -Dvideo=true -Dvenus=true
ninja -C build
ninja install

qemu

install qemu >= 9.2.0, at the time of writing ubuntu has not yet packaged it

Install build depdencies: https://wiki.qemu.org/Hosts/Linux

sudo apt-get install build-essential pip libslirp-dev slirp
sudo apt-get install git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev ninja-build
sudo apt-get install git-email
sudo apt-get install libaio-dev libbluetooth-dev libcapstone-dev libbrlapi-dev libbz2-dev
sudo apt-get install libcap-ng-dev libcurl4-gnutls-dev libgtk-3-dev
sudo apt-get install libibverbs-dev libjpeg8-dev libncurses5-dev libnuma-dev
sudo apt-get install librbd-dev librdmacm-dev
sudo apt-get install libsasl2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh-dev
sudo apt-get install libvde-dev libvdeplug-dev libvte-2.91-dev libxen-dev liblzo2-dev
sudo apt-get install valgrind xfslibs-dev 
sudo apt-get install libnfs-dev libiscsi-dev

build and run as described

virt-manager

-- work in progress --

Currently this is work in progress, so there is no option to add vulkan support in virt-manager. There are no fields to configure this. Also xml doesnt work, because libvirt doesn't know about these options either, so xml validation fails. There is however an option for QEMU command-line passthrough which bypasses the validation.

If you setup a default machine with 4G of memory, you can do this:

  <qemu:commandline>
    <qemu:arg value="-device"/>
    <qemu:arg value="virtio-vga-gl,hostmem=4G,blob=true,venus=true"/>
    <qemu:arg value="-object"/>
    <qemu:arg value="memory-backend-memfd,id=mem1,size=4G"/>
    <qemu:arg value="-machine"/>
    <qemu:arg value="memory-backend=mem1"/>
    <qemu:arg value="-vga"/>
    <qemu:arg value="none"/>
  </qemu:commandline>

Which gives this error:

qemu-system-x86_64: virgl could not be initialized: -1

Changing the number from 4G to 4194304k (same as memory) leds to this error:

qemu-system-x86_64: Spice: ../spice-0.15.2/server/red-qxl.cpp:435:spice_qxl_gl_scanout: condition `qxl_state->gl_draw_cookie == GL_DRAW_COOKIE_INVALID' failed

to be further investigated.

Hi @thesword53, could You please share necessary steps for VMWare? Thanks in advance

https://xanmod.org/ finally made 6.13 as stable, udmabuff included. Its alternative to building mainland.

Hi @thesword53, could You please share necessary steps for VMWare? Thanks in advance

I just installed VMware from AUR on the guest with venus enabled and I obviously enabled nested virtualization.

-display gtk,gl=on returns OpenGL is not supported by display backend 'gtk'

I get a lot of stutter When I play a game (ALOT!) almost all 3D games do it. But this guy (Yes I know it was before it was main stream but still) Was perfect:

https://www.youtube.com/watch?v=HmyQqrS09eo
https://www.youtube.com/watch?v=Vk6ux08UDuA

even this guy had less them me:

https://www.youtube.com/watch?v=icCxskFVapU

Does anyone know why this is happening?

I guess it is worth mentioning I have a AMD Radeon RX 6950 XT and need to give this patch in kernel 6.13.0 in drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c for vkcube or ... anything really to work(line numbers not accurate):

@@ -1143,8 +1143,10 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev,
	if (ret)
		return ret;

-	for (i = 0; i < ttm->num_pages; ++i)
+	for (i = 0; i < ttm->num_pages; ++i) {
		ttm->pages[i]->mapping = bdev->dev_mapping;
+		page_ref_inc(ttm->pages[i]);
+	}

	return 0;
}
------------------------------------------------------------------------
	if (ttm->page_flags & TTM_TT_FLAG_EXTERNAL)
		return;

-	for (i = 0; i < ttm->num_pages; ++i)
+	for (i = 0; i < ttm->num_pages; ++i) {
		ttm->pages[i]->mapping = NULL;
+		page_ref_dec(ttm->pages[i]);
+	}

And no newer kernels do not work (even with the patch)

I'm at my limit. I don't know if it's my hardware, my bios settings, the patch, I simply don't know. Any ideas?

EDIT: I'm on Linux mint if it helps.

EDIT2: MY GOD! I keep forgetting! Sorry if this is spamming you email. Here is my QEMU script.

/tools/virtualization/venus/qemu/build/qemu-system-x86_64 \
	-enable-kvm \
	-smp $CPU_CORES \
	-m $MEMORY \
	-hda $DISK \
	-audio pa,id=snd0,model=virtio,server=/run/user/1000/pulse/native \
	-overcommit mem-lock=off \
	-rtc base=utc \
	-serial mon:stdio \
	-device virtio-vga-gl,hostmem=$VRAM,blob=true,venus=true \
	-display gtk,gl=es,show-cursor=on \
	-object memory-backend-memfd,id=mem1,size=$MEMORY,share=on \
	-machine q35,accel=kvm,kernel-irqchip=split,memory-backend=mem1 \
	-net nic,model=virtio \
	-netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no \
	-device virtio-net-pci,netdev=mynet0 \
	-device virtio-balloon \
	-vga none \
	-net user,hostfwd=tcp::2222-:22 \
	-usb \
	-full-screen \
	-device usb-host,vendorid=$CONTROLERVID,productid=$CONTROLERPID \
	-kernel $KERNEL \
	-initrd $INITRD \
	-append "BOOT_IMAGE=$KERNEL root=/dev/sda2 rw nowatchdog nvme_load=YES zswap.enabled=0 loglevel=3 mitigations=off quiet splash" \
	-sandbox on \
	-d guest_errors \
	-boot c,menu=on \
	-cdrom $ISO

all variables are set.