Run Win11 on ARM in QEMU

README.md

When following this guide on a host not capable of native arm64 KVM, replace -M virt -cpu host -accel kvm with -M virt,virtualization=on -cpu max.

The reason for virtualization=on is that the Windows bootloader does an smc #0 PSCI call, but without EL2, QEMU's TCG does not handle those because PSCI is in HVC mode and such that instruction is treated as undefined. With KVM enabled, smc #0 is handled properly.

Workaround in QEMU for using TCG without virtualization=on:

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index b871350856..de11ff51d1 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -2056,7 +2056,7 @@ static void machvirt_init(MachineState *machine)
      */
     if (vms->secure && firmware_loaded) {
         vms->psci_conduit = QEMU_PSCI_CONDUIT_DISABLED;
-    } else if (vms->virt) {
+    } else if (1 || vms->virt) {
         vms->psci_conduit = QEMU_PSCI_CONDUIT_SMC;
     } else {
         vms->psci_conduit = QEMU_PSCI_CONDUIT_HVC;

1. Download the VM disk image from https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewarm64 and the latest virtio-win.iso from https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/. Build 22589 is known to work, while build 25226 is known not to work (unhandled exception during boot).
2. Perform the initial boot. For some reason performance is abysmal with the GTK UI, so use VNC: qemu-system-aarch64 -M virt -cpu host -accel kvm -m 2G -smp 2 -device ramfb -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device qemu-xhci -device usb-kbd -device usb-tablet -drive file=Windows11_InsiderPreview_Client_ARM64_en-us_22598.VHDX,format=vhdx,if=none,id=boot -device usb-storage,drive=boot,serial=boot -drive file=virtio-win.iso,media=cdrom,if=none,id=iso -device usb-storage,drive=iso -nic user,model=virtio-net-pci,mac=52:54:98:76:54:32 -vnc :0. On some earlier QEMU/kernel (?) versions, ramfb has some issues with invalidation (?) causing artifacts, you'll have to ignore them until virtio-gpu can be used. At least since build 25931, it's necessary to provide a network connection to finish the initial setup. The default QEMU MAC address will result in a false UUID match with some AutoPilot enrolled devices, forcing an "Intellek" domain login. To avoid that, use a different MAC address, e.g. 52:54:98:76:54:32 from the QEMU man page. To have a higher resolution with ramfb, enter the OVMF setup by pressing F10 during boot, change the preferred resolution to 1024x768 and reset.
3. This might take a while. If it appears stuck at "Getting ready" for 10min (give it a bit more time without KVM) or you get a BSOD (watchdog timeout), reset. If you get an error message "Windows installation cannot proceed", press Shift-F10 to open a cmd, run regedit and set HKEY_LOCAL_MACHINE\SYSTEM\Setup\Status\ChildCompletion\Setup to 3 and click on Ok to reboot.
4. When it asks for the initial setup (country, keyboard layout) press Shift-F10 to open a cmd, or (if it does not work) Ctrl-Shift-Esc to open the task manager and launch cmd with admin privs.
5. Run bcdedit /set TESTSIGNING ON to allow loading some drivers during boot. While virtio drivers are signed, they don't work without this for some reason.
6. Start explorer.exe, navigate to the virtio drivers in D: and install NetKVM, viogpudo, viostor and vioscsi by right clicking on the .inf file within the w11/ARM64 folders and selecting "Install". If it complains about driver signatures, navigate to the cert folder and install the certificate into the devices Trusted Publishers store.
7. Complete the setup as usual. To avoid using a Microsoft account, attempt to sign in as "Administrator" with some random string as password, then click "Next" on the "Oops, something went wrong" screen. If that no longer works, the oobe \bypassnro method is necessary.
8. Shut down Windows.
9. Now virtio devices can be used: qemu-system-aarch64 -M virt -cpu host -accel kvm -m 4G -smp 4 -device virtio-gpu-pci -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device qemu-xhci -device usb-kbd -device usb-tablet -drive file=Windows11_InsiderPreview_Client_ARM64_en-us_22598.VHDX,format=vhdx,if=none,id=boot -device virtio-scsi -device scsi-hd,drive=boot -nic user,model=virtio-net-pci,mac=52:54:98:76:54:32 -vnc :0. The first boot will take a while with a disabled display, be patient. If it does not work, try to boot with the previous QEMU command but also add -device virtio-gpu-pci and some virtio-scsi HD, to make sure the driver is properly selected.

openQA setup.md

openQA expects a resolution of 1024x768. The default is 1280x1024 and for some reason there's no option to change that in the UI, so it has to be done in the registry directly. This needs to be done with a very similar command line to actual QA jobs, otherwise the device doesn't match (based on PCI address).

1. Ctrl-Shift-Esc to open taskmgr
2. "Run new task", enter regedit and run it with admin privs
3. Navigate to HKLM/SYSTEM/CurrentControlSet/Control/GraphicsDrivers/Configuration/RHT*/00
4. Set PrimSurfSize.cx to 1024, PrimSurfSize.cy to 768 and Stride to 4096 (decimal!)
5. Reboot

To avoid having to add new needles, switch the desktop wallpaper to the one used by the existing win11 tests and change the terminal in the developer settings to "Windows Console Host". Also set a fixed background picture for the lock and sign in screens.

Windows on Arm has a different model for serial ports, the usual methods to access them no longer work. However, FTDI ported the old serial port bus to Arm and it works with the usb-serial device! Download https://www.ftdicommunity.com/index.php?topic=753.0 and extract it somewhere. The "QEMU USB SERIAL" device needs the driver assigned manually by using the device manager. To use it with openQA, pass -chardev ringbuf,id=usbserial,logfile=serial0,logappend=on -device usb-serial,chardev=usbserial. This uses a bit of a hack as it writes into the same file as the default serial port (pl011, not usable as COM port), but there's no other way as QEMU does not allow using a chardev from multiple devices. Native support for usb-serial in os-autoinst would be the right way. usb-serial also conflicts with -no-migratable, so QEMU_DISABLE_SNAPSHOTS=1 needs to be passed.

On Win11 build 25267 or newer, SMB guest authentication needs to be enabled (https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/guest-access-in-smb2-is-disabled-by-default), but it appears to be broken even with this, just differently...

1. Run gpedit.msc
2. Computer Configuration > Administrative Templates > Network > Lanman Workstation.
3. Enable insecure guest logons -> Enable

Not specific to Arm: To optimize the qcow2 size, start the VM with discard=unmap on the qcow2 block dev. In a powershell, run Optimize-Volume -Verbose -DriveLetter C -ReTrim and shut down. Run qemu-img convert -pc -O qcow2 for_cleanup.qcow2 Win11ArmBuild25931.qcow2 to discard the unmapped space and perform recompression.

Based on a win11 x86_64 (UEFI) test:

openqa-clone-job --within-instance https://openqa.opensuse.org --skip-chained-deps 3546891 _GROUP=38 ARCH=aarch64 HDD_1=win11_22598.qcow2 QEMUMACHINE=virt QEMUVGA= QEMU_VIDEO_DEVICE=virtio-gpu-pci UEFI_PFLASH_CODE=/usr/share/qemu/aavmf-aarch64-code.bin UEFI_PFLASH_VARS=/usr/share/qemu/aavmf-aarch64-vars.bin WORKER_CLASS=qemu_aarch64 HDDMODEL=scsi-hd QEMU_DISABLE_SNAPSHOTS=1 "QEMU_APPEND=chardev ringbuf,id=usbserial,logfile=serial0,logappend=on -device usb-serial,chardev=usbserial"

WSL2.md

It's also possible to run WSL2 inside the Windows VM. However, that uses Hyper-V and thus requires hardware virtualization. Nested virtualization is not widely supported by hardware yet and is also not implemented in Linux or QEMU, so software emulation is needed.

Just -M virt,virtualization=on -cpu max as used above in the non-KVM case will not work though: Windows no longer boots if the hypervisor is enabled. With WinDbg attached it complains about unexpected register values which indicate that it expects the CPU to have EL3.

Using https://github.com/ARM-software/arm-trusted-firmware in EL3 (following https://trustedfirmware-a.readthedocs.io/en/latest/plat/qemu.html) does not work though, it leads to Windows failing to initialize interrupts. Presumably some of the ATF initialization confuses the Windows kernel.

There's a way around ATF though: QEMU's -kernel option allows to have a CPU with EL3 but boot directly into EL2. To be able to use OVMF with that option (instead of -bios), it has to be built as Linux kernel image. A prebuilt QEMU_EFI.fd is available at https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/5312/QEMU-KERNEL-AARCH64/RELEASE_GCC/QEMU_EFI.fd.

For QEMU < 8.2.0 you either need to use -cpu neoverse-n1 or apply https://lists.nongnu.org/archive/html/qemu-devel/2023-09/msg03039.html, to avoid that accesses to HDFGRTR_EL2 trap to EL3 which does not exist -> stuck.

diff --git a/hw/arm/boot.c b/hw/arm/boot.c
index 720f22531a..24fa169060 100644
--- a/hw/arm/boot.c
+++ b/hw/arm/boot.c
@@ -761,6 +761,10 @@ static void do_cpu_reset(void *opaque)
                     if (cpu_isar_feature(aa64_hcx, cpu)) {
                         env->cp15.scr_el3 |= SCR_HXEN;
                     }
+                    if (cpu_isar_feature(aa64_fgt, cpu)) {
+                        env->cp15.scr_el3 |= SCR_FGTEN;
+                    }
+
                     /* AArch64 kernels never boot in secure mode */
                     assert(!info->secure_boot);
                     /* This hook is only supported for AArch32 currently:

QEMU < 8.1 is missing also SCR_HXEN and probably others, so make sure to use at least 8.1.0.

Full QEMU command with WSL2/Hyper-V support (tested with Build 25931):

qemu-system-aarch64 -M virt,virtualization=on,gic-version=3,secure=on -cpu neoverse-n1 -m 4G -smp 4 -device virtio-gpu-pci -kernel QEMU_EFI.fd -device qemu-xhci -device usb-kbd -device usb-tablet -drive file =Win11ArmBuild25931.qcow2,if=none,id=boot -device virtio-scsi -device scsi-hd,drive=boot,serial=hd0 -nic user,model=virtio-net-pci,mac=52:54:98:76:54:32 -vnc 0

Windows does not have any drivers for virtio-gpu out of the box

Assuming the VM's hard drive has Windows and the virtio-gpu driver correctly installed, is it possible to boot without ramfb?

Yes, that's what step 9 does.