Linux Desktop on Apple Silicon in Practice

README.en.md

Linux Desktop on Apple Silicon in Practice

I bought M1 MacBook Air. It is the fastest computer I have, and I have been a GNOME/GNU/Linux user for long time. It is obvious conclusion that I need practical Linux desktop environment on Apple Silicon.

Fortunately, Linux already works on Apple Silicon/M1. But how practical is it?

• Two native ports exist.
  • Corellium. It is obsolete. https://corellium.com/blog/linux-m1
  • Asahi Linux. It is greatly improving and may be usable for certain workloads, but it lacks e.g. graphics acceleration for now. https://asahilinux.org
• QEMU can run code on CPU natively. But what about GPU? Unfortunately, QEMU is also not optimized so much for macOS.

As I needed Linux desktop right now, I decided to hack QEMU. The most difficult challenge is obviously accelerated graphics, but there is Virgil 3D; a bridge to expose host OpenGL to the guest. https://virgil3d.github.io

It unfortunately didn't work on macOS host. So I just made it work. That's it. Here is a video demonstrating OpenGL on Linux on Apple Silicon/M1:

https://www.youtube.com/watch?v=k0bVlVQU2JQ&list=PLesZxBYUPr3wdU3sONUv4Q7UDOg1Dn_Ue&index=4

Modifications

QEMU

ui/cocoa

• Added OpenGL support.
• Enforced pixel by pixel display.
• Added cursor composition.
• Improved key mappings (e.g. Japanese IME keys, 2021-06-17)

hw/block

• File locking on macOS is fixed. (2021-07-07, Add file.locking=on to drive to prevent drive breakage in case you concurrently launch the same virtual machine by mistake.)

coreaudio

• Fix device change (2022-02-26)

Virgil 3D renderer

Improved OpenGL ES support.

Do It Yourself

@knazarov's Homebre Formulae

It is independently maintained so may be a bit older, but you may still find it useful.

https://github.com/knazarov/homebrew-qemu-virgl

Setup

1. Open a terminal.

2. Install GLib, Meson, Pixman, pkg-config and spice-protocol with Homebrew.

brew install glib meson pixman pkg-config spice-protocol

3. Make a empty directory and change the working directory to it.

4.

curl -L https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807ec5a1bc5/raw/2e1e5572cec78625df452b16b0ee924ef86aed0d/run.sh | bash -

5.

bin/qemu-img create var/virtio.raw 64G

It doesn't consume the physical space until it has data, so you can make the image very large. However, you will see odd behavior if you try to write data more than the physical disk allows.

6.

curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/40/Silverblue/aarch64/iso/Fedora-Silverblue-ostree-aarch64-40-1.14.iso

8.

./run -cdrom Fedora-Silverblue-ostree-aarch64-40-1.14.iso

Proceed the installation process, and now you can run Fedora by executing ./run.

Note: you won't get keyboard to work before Linux boots because TianoCore, the UEFI firmware does not support virtio-keyboard used in this configuration.

Updating

Just download the latest run.sh and execute it in your workspace directory.

Choosing OpenGL profile

Edit run.

• gl=off will disable Virgil 3D GPU. Most stable but laggy.
• gl=core will enable OpenGL.framework. Unstable.
• gl=es will enable ANGLE. Stable and fast.

Upstreaming

Upstreaming is in progress. Hopefully the features I implemented will work just by running brew install qemu in the future.

• Epoxy: anholt/libepoxy#239
• QEMU: https://patchew.org/search?q=project%3AQEMU+from%3Aakihiko.odaki%40gmail.com
• Virgil 3D renderer: https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests?scope=all&utf8=✓&state=all&author_username=akihiko.odaki

Some insights

QEMU

This QEMU modification is not secure. The graphics acceleration code lives in the same process with everything else of the virtual machine and it is huge; the graphics stack involves LLVM for shader compilation, and a simple bug in the stack can lead to complete takeover of the guest.

vhost-user-gpu provides graphics acceleration isolation, but it needs modifications to run outside Linux because:

• historically, vhost-user is a re-implementation of Linux kernel's vhost interface, and it relies on kernel headers for interface definitions and
• vhost-user uses eventfd which is only available on Linux.

It shouldn't be difficult, but I'm satisfied even without process isolation so I don't. The graphics acceleration process would be still shared and it would remain possible that one graphical process exploit leads to disclosure of the entire graphics output anyway.

Linux desktop on Apple Silicon/M1 in general

As I described here, such a virtualization software is practical and efficient approach to run Linux desktop. The performance overhead is also acceptable for daily use, and it even provides better integration of Linux and macOS. For example, you can switch macOS and Linux with three-finger gesture on trackpad. You can use VirtFS.

However, there are complexities that such a virtualization adds. It basically means sharing one hardware with two systems, so you have to allocate the resource properly or it ends up with bad user experience. The allocation problem happens everywhere (HID like keyboard, computing resource like CPU, power management, etc.). This approach is efficient but not the best.

In long term, native Linux port is the best option. Asahi Linux is promising and you may find it favorable than my modified QEMU for your use case even today.

README.ja.md

Apple Silicon で実践Linuxデスクトップ

M1 MacBook Airを買いました. これは私が持ってる一番速いコンピュータで, 私は長年の GNOME/GNU/Linux ユーザーでもあります. ここから, Apple Silicon でLinux デスクトップ環境が必要であるという明らかな結論が導かれます.

幸いにも, Linux は Apple Silicon で既に動作します. しかし 実用性 はどうでしょうか?

• 2つのネイティブの移植があります.
  • Corellium. これは古くなっています. https://corellium.com/blog/linux-m1
  • Asahi Linux. 大幅に改善されており, 場合によっては実用的と言えます. しかし, 今のところグラフィックスアクセラレーションなどが欠けています. https://asahilinux.org
• QEMU は CPU上でコードをネイティブ動作させることが可能です. しかし GPU についてはどうでしょう? また, 残念ながら, QEMU は macOS 向けにあまり最適化されていません.

私は Linux デスクトップが直ちに必要だったので, QEMU をハックすることにしました. 最大の困難は当然グラフィックスアクセラレーションですが, このために用いることができる, Virgil 3D という, ホストの OpenGL をゲストに見せるブリッジが存在します. https://virgil3d.github.io

残念ながらこれは macOS ホスト上で動きませんでした. そういうわけで 動くようにしました. 以上! 以下は Apple Silicon/M1 上での OpenGL を実演する動画です.

https://www.youtube.com/watch?v=k0bVlVQU2JQ&list=PLesZxBYUPr3wdU3sONUv4Q7UDOg1Dn_Ue&index=4

改変

QEMU

ui/cocoa

• OpenGL サポートを追加.
• Pixel by pixel 表示を実装.
• カーソルの合成を追加.
• キーマッピングを改善 (日本語 IME キーなど, 2021-06-17)

hw/block

• macOS でのファイルロッキングを修正. (2021-07-07, 間違って同じ仮想マシンを 並行に起動してしまった場合にディスクが壊れるのを防ぐためには drive に file.locking=on を追加してください.)

hvf

• 最新の Linux のために AArch64 ID レジスタを修正 (@agraf, 2022-02-07)

coreaudio

• デバイスの変更処理を修正 (2022-02-26)

Virgil 3D renderer

OpenGL ES サポートを改善

Do It Yourself

@knazarov の Homebrew formulae

独自に保守されているため少し古いかもしれませんが, 便利かもしれません.

https://github.com/knazarov/homebrew-qemu-virgl

セットアップ

1. ターミナルを開く.

2. GLib, Meson, Pixman, pkg-config, そして spice-protocol を Homebrew でインストールする.

brew install glib meson pixman pkg-config spice-protocol

3. 空のディレクトリを作って working directory をそれに変更する.

4.

curl -L https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807ec5a1bc5/raw/2e1e5572cec78625df452b16b0ee924ef86aed0d/run.sh | bash -

5.

bin/qemu-img create var/virtio.raw 64G

データが記録されるまで物理領域を消費しないため, イメージをかなり大きくできます. ただし, 物理ディスクが許容する以上のデータを書き込むとおかしな挙動が現れます.

6.

curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/40/Silverblue/aarch64/iso/Fedora-Silverblue-ostree-aarch64-40-1.14.iso

8.

./run -cdrom Fedora-Silverblue-ostree-aarch64-40-1.14.iso

インストールプロセスを進めてください. ./run を実行することで Fedora を起動できるようになるはずです.

注: Linux が起動するまでキーボードは利用できません。これは、この構成で利用している virtio-keyboard に UEFI ファームウェアの TianoCore が対応していないためです。

更新

単に最新の run.sh をダウンロードしてワークスペースディレクトリで実行して ください.

OpenGL プロファイルを選択する

run を編集してください.

• gl=off は Virgil 3D GPU を無効にします. 安定していますがラグいです.
• gl=core は OpenGL.framework を有効にします. 不安定です.
• gl=es は ANGLE を有効にします. 安定していて速いです.

アップストリーミング

アップストリーミングが進行中です. 願わくば将来は私が実装した機能が brew install qemu とするだけで動作するようになるはずです.

• Epoxy: anholt/libepoxy#239
• QEMU: https://patchew.org/search?q=project%3AQEMU+from%3Aakihiko.odaki%40gmail.com
• Virgil 3D renderer: https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests?scope=all&utf8=✓&state=all&author_username=akihiko.odaki

考察

QEMU

この QEMU の改変は セキュアではありません . グラフィックスアクセラレーションのコードが仮想マシンの他のあらゆるもの全てと同じプロセスに存在していて, そのコードは巨大です. グラフィックススタックはシェーダコンパイルのために LLVM を含んでおり, その中の簡単な不具合1つでゲストの完全な乗っ取りが可能です.

vhost-user-gpu はグラフィックスアクセラレーションの分離を提供しますが, Linux 以外で動作するように修正が必要です.

• 歴史的には, vhost-user は Linux カーネルの vhost インターフェイスの再実装となっていて, インターフェイス定義のためにカーネルヘッダに依存しています.
• vhost-user は Linux でしか利用できない eventfd を利用しています.

これは難しくないでしょうが, プロセス分離がなくても満足してるのでやりません. やってもグラフィックスアクセラレーションプロセスは共有されたままになるでしょうし, 単独のグラフィカルプロセスの攻撃がグラフィックス出力全体の暴露につながるのは変わりないでしょう.

Apple Silicon/M1 上の Linux デスクトップ一般について

先に説明したとおり, このような仮想化ソフトウェアは Linux デスクトップを実行する 実用的で効率的な方法です. 性能上のオーバーヘッドも日常的な利用には許容できる範囲で, Linux と macOS のよりよい統合を提供しさえします. 例えば, macOS と Linux をトラックパッド上の3本指ジェスチャーで切り替えられます. VirtFS も使えます.

しかし, 仮想化による複雑さもあります. 要は1つのハードウェアを2つのシステムで 共有することになるので, 資源を適切に割り当てなければユーザー体験を悪化させます. この問題はあらゆる場面でおきます (キーボードのような HID, CPU のような計算資源, 電力管理などなど). この方法は 効率的 ですが 最善 ではありません.

長期的には ネイティブの Linux 移植が最善でしょう. Asahi Linux は期待できます. 今でも利用法によっては私が改変した QEMU よりよいかもしれません.

run.sh

set -eux
mkdir -p depot_tools build/qemu source/angle source/libepoxy source/virglrenderer source/qemu var
git -C depot_tools init
git -C depot_tools fetch https://chromium.googlesource.com/chromium/tools/depot_tools e1385296c4ab4c7ee0a809676635b52d1df23b87
git -C depot_tools checkout FETCH_HEAD
git -C source/angle init
git -C source/angle fetch https://chromium.googlesource.com/angle/angle 5d4df51d1d7d6a290d54111527a4798f10c7ca3c
git -C source/angle checkout FETCH_HEAD
git -C source/libepoxy init
git -C source/libepoxy fetch https://github.com/akihikodaki/libepoxy.git macos
git -C source/libepoxy checkout FETCH_HEAD
git -C source/virglrenderer init
git -C source/virglrenderer fetch https://github.com/akihikodaki/virglrenderer.git macos
git -C source/virglrenderer checkout FETCH_HEAD
git -C source/qemu init
git -C source/qemu fetch https://github.com/akihikodaki/qemu.git macos
git -C source/qemu checkout FETCH_HEAD
export DEPOT_TOOLS_UPDATE=0
export PATH="$PWD/depot_tools:$PATH"
cd source/angle
scripts/bootstrap.py
gclient sync -D
gn gen --args=is_debug=false ../../build/angle
cd ../..
ninja -C build/angle
[ -e build/libepoxy/meson-info ] || meson setup "-Dc_args=-I$PWD/source/angle/include" -Degl=yes -Dx11=false "--prefix=$PWD" build/libepoxy source/libepoxy
meson install -C build/libepoxy
[ -e build/virglrenderer/meson-info ] || meson setup "-Dc_args=-I$PWD/source/angle/include" "--pkg-config-path=$PWD/lib/pkgconfig" "--prefix=$PWD" build/virglrenderer source/virglrenderer
meson install -C build/virglrenderer
cd build/qemu
PKG_CONFIG_PATH="$PWD/../../lib/pkgconfig" ../../source/qemu/configure "--extra-cflags=-I$PWD/../../source/angle/include" "--extra-ldflags=-L$PWD/../angle" "--prefix=$PWD/../.."
make "-j$(getconf _NPROCESSORS_ONLN)" install
[ -e ../../var/edk2-arm-vars.fd ] || cp pc-bios/edk2-arm-vars.fd ../../var
cd ../..
cat > run <<'EOF'
#!/bin/bash
d="$(dirname "${BASH_SOURCE[0]}")"
exec sudo DYLD_FALLBACK_LIBRARY_PATH="$d/build/angle:$d/lib" "$d/bin/qemu-system-aarch64" -machine virt,accel=hvf -cpu host -smp "$(getconf _NPROCESSORS_ONLN)" -m 4G -device pcie-root-port,id=pcie -device virtio-sound-pci,addr=0x0.0x0,bus=pcie,multifunction=on,audiodev=audio,streams=1 -device virtio-gpu-gl-pci,addr=0x0.0x1,bus=pcie -device virtio-keyboard-pci,addr=0x0.0x2,bus=pcie -device virtio-net-pci,addr=0x0.0x3,bus=pcie,netdev=net -device virtio-rng-pci,addr=0x0.0x4,bus=pcie -display cocoa,gl=es -drive "if=pflash,format=raw,file=$d/share/qemu/edk2-aarch64-code.fd,readonly=on" -drive "if=pflash,format=raw,file=$d/var/edk2-arm-vars.fd" -drive "id=virtio,if=none,format=raw,file=$d/var/virtio.raw,discard=on" -device virtio-blk-pci,addr=0x0.0x5,backend_defaults=on,bus=pcie,drive=virtio -audiodev coreaudio,id=audio,out.fixed-settings=false -netdev vmnet-shared,id=net -chardev qemu-vdagent,id=spice,name=vdagent,clipboard=on -device virtio-serial-pci,addr=0x0.0x6,bus=pcie -device virtserialport,chardev=spice,name=com.redhat.spice.0 -run-with user="$(id -u):$(id -g)" "$@"
EOF
chmod a+x run

@ZeppLu You are absolutely right and @akihikodaki pointed out the lack of support under android but I guess I was too carried out to listen.

Here is a proper test from ubuntu noble:

Big Thanks.

While trying to find ways to speed up host-guest communication, through some (admittedly not very scientific) iperf tests, I noticed that 1. TCP was faster if the Wi-Fi is turned on, and the host can access the internet (ie, just turning it on and not connecting to a network is not enough); and 2. that communication seems capped at 1Gib/s. I think 2 is expected (this is just the maximum throughput of the underlying vmnet.framework), but I'm not sure about 1.

I tried building again, had to remove everything and start from scratch. the resulting binaries won't work. It won't boot my vm, is this known?

@j03ll0b0 What do you mean "won't work"? What errors are you getting?

Found the issue. I profiled qemu and realized that the shader compilation freezes the process for hundreds of milliseconds, leading to the observed lag within the machine. I recompile angle with the metal backend, as that is supposed to speed up shader compile times compared to apples opengl implementation, which it did. There is still notable lag when opening new windows within gnome. I tracked that down to dozens of shaders being compiled (and recompile) everytime a gtk4 window opens. I am now adding some hacky caching to virglrenderer to cache the compiled shaders, so that they do not need to be recompile every time. This removes any lag from opening windows for me.
I'll share a patch once I get this working correctly.

Previous text:
~~This thing works great for me with one exception: Everytime i am opening a new window in the guest, the screen freezes for about 3 or 4 seconds. This also happens if I open a second window within the same app (e.g. gnome console kgx). It seems to happen with recent gnome 46.4 as well as with an older version like ~43. Also mesa 24.0 to 24.2 do not seem to make a difference. Do you have any pointers on how i can start debugging this? Can I somehow get a virgl trace from the guest, or a virglrender trace from the host?~~

If I run:

Macintosh:~ vagrant$  glxgears -info
No matching pixelformats found, perhaps try setting LIBGL_ALLOW_SOFTWARE=true
Abort trap: 6

@akihikodaki So I can only run glxgears through software but not virgl ?

LIBGL_ALLOW_SOFTWARE=true glxgears -info
GL_RENDERER   = Apple Software Renderer
GL_VERSION    = 2.1 APPLE-9.6.5
GL_VENDOR     = Apple Computer, Inc.
GL_EXTENSIONS = GL_ARB_color_buffer_float GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_depth_texture GL_ARB_draw_buffers GL_ARB_draw_elements_base_vertex GL_ARB_draw_instanced GL_ARB_fragment_program GL_ARB_fragment_program_shadow GL_ARB_fragment_shader GL_ARB_framebuffer_object GL_ARB_framebuffer_sRGB GL_ARB_half_float_pixel GL_ARB_half_float_vertex GL_ARB_imaging GL_ARB_instanced_arrays GL_ARB_multisample GL_ARB_multitexture GL_ARB_occlusion_query GL_ARB_pixel_buffer_object GL_ARB_point_parameters GL_ARB_point_sprite GL_ARB_provoking_vertex GL_ARB_seamless_cube_map GL_ARB_shader_objects GL_ARB_shader_texture_lod GL_ARB_shading_language_100 GL_ARB_shadow GL_ARB_shadow_ambient GL_ARB_sync GL_ARB_texture_border_clamp GL_ARB_texture_compression GL_ARB_texture_compression_rgtc GL_ARB_texture_cube_map GL_ARB_texture_env_add GL_ARB_texture_env_combine GL_ARB_texture_env_crossbar GL_ARB_texture_env_dot3 GL_ARB_texture_float GL_ARB_texture_mirrored_repeat GL_ARB_texture_non_power_of_two GL_ARB_texture_rectangle GL_ARB_texture_rg GL_ARB_transpose_matrix GL_ARB_vertex_array_bgra GL_ARB_vertex_blend GL_ARB_vertex_buffer_object GL_ARB_vertex_program GL_ARB_vertex_shader GL_ARB_window_pos GL_EXT_abgr GL_EXT_bgra GL_EXT_bindable_uniform GL_EXT_blend_color GL_EXT_blend_equation_separate GL_EXT_blend_func_separate GL_EXT_blend_minmax GL_EXT_blend_subtract GL_EXT_clip_volume_hint GL_EXT_debug_label GL_EXT_debug_marker GL_EXT_depth_bounds_test GL_EXT_draw_buffers2 GL_EXT_draw_range_elements GL_EXT_fog_coord GL_EXT_framebuffer_blit GL_EXT_framebuffer_multisample GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_framebuffer_object GL_EXT_framebuffer_sRGB GL_EXT_geometry_shader4 GL_EXT_gpu_program_parameters GL_EXT_gpu_shader4 GL_EXT_multi_draw_arrays GL_EXT_packed_depth_stencil GL_EXT_packed_float GL_EXT_provoking_vertex GL_EXT_rescale_normal GL_EXT_secondary_color GL_EXT_separate_specular_color GL_EXT_shadow_funcs GL_EXT_stencil_two_side GL_EXT_stencil_wrap GL_EXT_texture_array GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_s3tc GL_EXT_texture_env_add GL_EXT_texture_filter_anisotropic GL_EXT_texture_integer GL_EXT_texture_lod_bias GL_EXT_texture_mirror_clamp GL_EXT_texture_rectangle GL_EXT_texture_shared_exponent GL_EXT_texture_sRGB GL_EXT_texture_sRGB_decode GL_EXT_timer_query GL_EXT_transform_feedback GL_EXT_vertex_array_bgra GL_APPLE_aux_depth_stencil GL_APPLE_client_storage GL_APPLE_element_array GL_APPLE_fence GL_APPLE_float_pixels GL_APPLE_flush_buffer_range GL_APPLE_flush_render GL_APPLE_packed_pixels GL_APPLE_pixel_buffer GL_APPLE_rgb_422 GL_APPLE_row_bytes GL_APPLE_specular_vector GL_APPLE_texture_range GL_APPLE_transform_hint GL_APPLE_vertex_array_object GL_APPLE_vertex_array_range GL_APPLE_vertex_point_size GL_APPLE_vertex_program_evaluators GL_APPLE_ycbcr_422 GL_ATI_separate_stencil GL_ATI_texture_compression_3dc GL_ATI_texture_env_combine3 GL_ATI_texture_float GL_ATI_texture_mirror_once GL_IBM_rasterpos_clip GL_NV_blend_square GL_NV_conditional_render GL_NV_depth_clamp GL_NV_fog_distance GL_NV_light_max_exponent GL_NV_texgen_reflection GL_NV_texture_barrier GL_SGIS_generate_mipmap GL_SGIS_texture_edge_clamp GL_SGIS_texture_lod 
VisualID 107, 0x6b
1485 frames in 5.0 seconds = 296.410 FPS
1276 frames in 5.0 seconds = 254.867 FPS
1419 frames in 5.0 seconds = 283.708 FPS
2100 frames in 5.0 seconds = 419.879 FPS
^C

I have increased the vRam in macOS to 512 MB as explained here
And Now I get up to 2000 FPS :
https://streamable.com/pmsb7b