Run a full linux desktop in a container

linux-full-desktop-container.md

In the following gist I'm going to guide you through the process of installing and booting an entire linux distribution with full desktop environment just like you would have with a classical VM, but with much better performance and much worse isolation :)

The reason why I did this was mainly because it's cool, but also to test new distros with decent graphics performance without actually booting them on my PC.

If you "try this at home" just keep in mind a container is not as secure as a VM, and some of the option we're going to explore will weaken container isolation from "a bit risky" to "totally unsafe" depending on what you choose.

Also, we're going to use systemd-nspawn for containers as it's probably the best fit for our use case and can also boot any linux partition without needing to prepare an apposite container image.

Less go!

First off, you will need to...

Bootstrap a linux container

There are at least three options I can think of:

1. Bootstrap the distro of your choice in a folder with the appropriate tool
2. Boot and install the distro of your choice in a Virtualbox VM, then extract the VM filesystem to a folder
3. Just boot a distro from another partition

Let's go with the former. For this example I chose Ubuntu 22.04 but you can install anything you like (Arch, Fedora, Ubuntu, Debian, ...) - just have a look at man systemd-nspawn for more details.

Bootstrap Ubuntu 22.04

1. Install debootstrap and schroot (the name of some packages might change from distro to distro)
2. mkdir ubuntu2204
3. sudo debootstrap --arch amd64 jammy ubuntu2204/ http://archive.ubuntu.com/ubuntu/
4. Chroot into the system: sudo systemd-nspawn -D ubuntu2204
5. Create your user: adduser username
6. Add your user to the following groups: sudo usermod -aG video,render,sudo username
7. Name resolution is probably broken due to systemd-resolved that can't run at this time, so mv /etc/resolv.conf{,.old} && echo "nameserver 8.8.8.8" > /etc/resolv.conf - of course you can change this to your preferred DNS.
8. Configure lang and locale: dpkg-reconfigure locales && apt install language-pack-it && locale-gen
9. Remove acpid because of an Ubuntu 22.04 bug.. apt remove acpid
10. Then exit the container

Time to install GPU Drivers

You will need to install your host gpu drivers to get good performance.

0. Boot the container: sudo systemd-nspawn -D ubuntu2204/ -b --register=no --keep-unit --bind=/dev/dri
1. Install some deps: sudo apt install libglvnd-dev pkg-config

If you're an nvidia user like me, you'll probably be better off installing the same driver version you have on your host system from the nvidia website and do not install kernel modules: sudo bash NVIDIA-Linux-*.run --no-kernel-modules will do.

If you're an AMD or INTEL user, I don't know what the procedure will be, but I'm going to guess it'll just work if you install your drivers like you normally do :)

Configure the container for GPU accelerated graphics

Install VirtualGL

You could skip this step depending on what configuration you end up setting to your container, but it's strongly recommended to install VirtualGL as it's going to be useful to get 3D graphics out of your container in the more "safe" configuration.

0. Boot into the container:

sudo systemd-nspawn -D ubuntu2204/ -b --register=no --keep-unit --bind=/dev/dri

1. Install dependencies and useful stuff:

sudo add-apt-repository universe
sudo apt install libegl1-mesa mesa-utils mesa-utils-bin

2. Install virtualgl from sourceforge (as it's probably not going to be in your repos, unless you use arch btw)

wget https://deac-ams.dl.sourceforge.net/project/virtualgl/3.0.1/virtualgl_3.0.1_amd64.deb
sudo dpkg -i virtualgl*.deb

3. Configure VirtualGL:

sudo vglserver_config

   				- Press 1 to Configure server for use with VirtualGL
   				  Answer 'Yes' to Restrict 3D X server access to vglusers group
   				  Answer 'Yes' to Restrict framebuffer device access to vglusers group
   				  Answer 'Yes' to Disable XTEST extension
   				  Press X to Exit

4. Add the user to the vglusers group.

sudo usermod -aG vglusers username

Last bits of config

0. Add this to your /etc/environment file:

PULSE_SERVER=tcp:127.0.0.1    # to hear audio from the container
DISPLAY=:12                   # must match your Xephyr display id as we'll see later

1. If you're an NVIDIA user, also add those env. variables to for the correct renderer:

__NV_PRIME_RENDER_OFFLOAD=1
__GLX_VENDOR_LIBRARY_NAME=nvidia
__VK_LAYER_NV_optimus=NVIDIA_only
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json

If you are an AMD or Intel user you'll probably need to do something similar but I can't test that for you. Have a look at this page for more info and let me know what works for you!

2. Now you can sudo reboot the container

Run the container

Now to the fun part. Here you have multiple options from "relatively safe" with acceptable performance to "very unsafe" with good performance. From my benchmarks, there is a ~30% gap in performance between the safer and the riskiest option.

The "safer" option

Is to run everything in a Xephyr window, not sharing memory or resources with the container directly. Performance is still quite good, but on the lower end of what you can get if you risk more as we'll see later. Still, I think this should be the preferred option for most people.

On the host

1. Run a Xephyr (X11 windowed server) session on the display you previously set: Xephyr :12 -screen 1920x1080 &
2. Optionally give anonymous local access to your pulse/pipewire audio server pactl load-module module-native-protocol-tcp auth-anonymous=1 auth-ip-acl="127.0.0.1" (not recommended if you don't need it)
3. Boot the container sudo systemd-nspawn -D ubuntu2204 -b --register=no --keep-unit

On the container (guest)

1. make sure you've disabled X11 shared memory as it won't work:

export QT_X11_NO_MITSHM=1

2. launch the Desktop Environment

vlgrun -d :12 gnome-session

PS: You can add QT_X11_NO_MITSHM=1 to your /etc/environment to make this permanent across reboots.

The "hazardous" option

There are a few middleground options between this solution and the safer one, but I'm not sure they really make much sense since we're breaking container isolation anyways. Needless to say you'll need to be very sure of what you're running when using this method, as it will be just as risky as running on your own machine and OS from a security perspective.

So... to get the most out of your container you'll need to share memory and devices directly with it. SYSTEMD_NSPAWN_SHARE_NS_IPC is a dangerous (and therefore undocumented?) environment variable for systemd-nspawn which will let the host and the container share IPC namespace. When using this, make sure QT_X11_NO_MITSHM=0 is set. Last thing you need to do is to share the video card with the container for EGL access. Wrapping everything up, we get..

On the host

pactl load-module module-native-protocol-tcp auth-anonymous=1 auth-ip-acl="127.0.0.1"  # for audio

Xephyr :12 -screen 1920x1080 &

SYSTEMD_NSPAWN_SHARE_NS_IPC=1 sudo -E systemd-nspawn -D ubuntu2204/ -b --register=no --keep-unit --bind=/dev/shm --bind=/dev/dri

On the container

export QT_X11_NO_MITSHM=0

vglrun -d /dev/dri/card0 gnome-session  # for EGL

OR

vglrun -d :12 gnome-session  # for GLX

OR

vglrun -d :0 application  # to run an application in seamless mode with the host*

*To be fair this shouldn't work unless you do xhost +local: which is a bad idea, or via a proper authentication which is still not a great idea.

I hope I didn't forget anything important but let me know if so. I'll leave you with a couple

Useful links

You might want to read to get a better feeling of what you're doing:

Linux graphics stack: https://blogs.igalia.com/itoral/2014/07/29/a-brief-introduction-to-the-linux-graphics-stack/

Linux graphics stack and drivers: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/inside-linux-graphics-paper.pdf

Diving into Mesa: https://blogs.igalia.com/itoral/2014/08/08/diving-into-mesa/

VirtualGL background: https://virtualgl.org/About/Background

VirtualGL documentation: https://rawcdn.githack.com/VirtualGL/virtualgl/3.0.1/doc/index.html

LD_PRELOAD: https://www.baeldung.com/linux/ld_preload-trick-what-is

Hi! I've followed your guide with Debian 11 as guest and Debian Testing as host but it seems that something's broken. glxinfo reports the following (regardless of whether it's run by vglrun or not, in both the safe and unsafe setups):

$ glxinfo 
name of display: :12
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  150 (GLX)
  Minor opcode of failed request:  24 (X_GLXCreateNewContext)
  Value in failed request:  0x0
  Serial number of failed request:  50
  Current serial number in output stream:  51

$ vglrun -d /dev/dri/card0 glxinfo
name of display: :12
[VGL] ERROR: in init3D--
[VGL]    197: No EGL devices found

$ env | grep -E 'NV_|nvidia|DISPLAY'
DISPLAY=:12
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json
__GLX_VENDOR_LIBRARY_NAME=nvidia
__VK_LAYER_NV_optimus=NVIDIA_only
__NV_PRIME_RENDER_OFFLOAD=1

Do you have an idea about what's going on?

But are you running Xephyr on display 12?

Yep. Running glxgears without setting nvidia as the GLX vendor name works, but only because it uses LLVMpipe (CPU rendering)

Edit: maybe Debian 11 is too old (even though the nvidia drivers are up to date)? I'll try Ubuntu 22.04 as a guest too.

Also check that host and guest driver versions match

I've retried with both Debian Testing and Ubuntu 22.04 as guests and it still gives the same error I mentioned earlier (I even changed the physical host)... I'm using the Nvidia driver version 470.141.03 (yes, on both the host and guest; I've even tried installing the driver package from apt, but it did not help). I'll try with the newer 510.85.02 drivers. On which host distros did you try this?

Edit: it doesn't work even with the 510.85.02 drivers :'(

Hi! I'm having some trouble while doing this. If I try to run a gnome-session it says "Terminated" and nothing happens, so I switched to xfce4 but I can run it only with EGL backend (and it render at 60Hz instead of 144Hz). GLX backend only worked the very first time I run it (but I saw that everything was rendered at 144Hz refresh). Do you why this happened and how I can solve it? (I'm trying to have all the container and programs rendered at 144Hz). Thanks!

Edit: when I try the GLX backend it return that: Xlib: extension "NV-GLX" missing on display ":12".