Emulating Raspberry Pi 4 with Qemu

RaspberryPi4-qemu.md

Emulating Raspberry Pi 4 with Qemu

Just a quick update before we dive in: what we're actually doing here is running Raspberry Pi OS (64-bit) on a QEMU virtual ARM setup. This isn't full-blown hardware emulation of the Raspberry Pi 4, but more about creating a virtual environment for the OS. It doesn't mimic all the specific hardware features of the Pi 4, but it's pretty useful and great for general testing. I turned to this solution mainly to extract a modified sysroot from the Raspberry Pi OS, something not readily available in other resources. For those looking into detailed emulation of the actual Raspberry Pi 4's hardware in QEMU, check out this link for the latest updates: https://gitlab.com/qemu-project/qemu/-/issues/1208.

Hope it helps! :D

Shortcomings: No GUI yet, only console.

Steps

1. Download Raspberry Pi OS (64-bit) from Raspberry Pi operating system images.
   Here we downloaded Raspberry Pi OS (64-bit) with desktop, Kernel version: 6.1, Debian version: 11 (bullseye), Release date: May 3rd 2023, named 2023-05-03-raspios-bullseye-arm64.img. We put it in /home/mydir.
2. Install the required packages on your host system:

   $ # Cross compilers for arm64
   $ sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
   
   $ # Qemu itself
   $ sudo apt install qemu qemubuilder qemu-system-gui qemu-system-arm qemu-utils \
       qemu-system-data qemu-system

3. Build the Linux kernel for qemu arm64 (You can download the kernel from Kernel.org):

   $ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.34.tar.xz
   $ tar xvJf linux-6.1.34.tar.xz
   $ cd linux-6.1.34
   
   $ # create a .config file
   $ ARCH=arm64 CROSS_COMPILE=/bin/aarch64-linux-gnu- make defconfig
   $ # Use the kvm_guest config as the base defconfig, which is suitable for qemu
   $ ARCH=arm64 CROSS_COMPILE=/bin/aarch64-linux-gnu- make kvm_guest.config
   $ # Build the kernel
   $ ARCH=arm64 CROSS_COMPILE=/bin/aarch64-linux-gnu- make -j8
   
   $ cp arch/arm64/boot/Image /home/mydir

4. Mount the image for enabling ssh and configuring username and password:
   1. Get the correct offset value with the help of fdisk utility:

      $ fdisk -l 2023-05-03-raspios-bullseye-arm64.img
      Disk 2023-05-03-raspios-bullseye-arm64.img: 4.11 GiB, 4412407808 bytes, 8617984 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 512 bytes
      I/O size (minimum/optimal): 512 bytes / 512 bytes
      Disklabel type: dos
      Disk identifier: 0x3e247b30
      Device                                 Boot  Start     End Sectors  Size Id Type
      2023-05-03-raspios-bullseye-arm64.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
      2023-05-03-raspios-bullseye-arm64.img2      532480 8617983 8085504  3.9G 83 Linux

      As we can see, we have two partitions inside the downloaded image. The first device (partition) is the bootable partition, and the second one is the root filesystem. The first partition is what will be mounted as /boot in Raspberry Pi, and this is where we'll need to create some files.
      Obtain the correct offset of the first device by multiplying the start of the first partition (here 8192) by the sector size (here 512). Here it will be calculated as 8192 * 512 = 4194304
   2. Mount the image in /mnt/rpi directory:

      $ sudo mkdir /mnt/rpi
      $ sudo mount -o loop,offset=4194304 2023-05-03-raspios-bullseye-arm64.img /mnt/rpi

   3. Create a file named ssh to enable ssh:

      $ cd /mnt/rpi
      $ sudo touch ssh

   4. Additionally, create a file named userconf.txt in the same directory and put your desired username and password there, like <username>:<hashed-password> (might be better to leave the username as pi). This will be your default credentials:

      $ openssl passwd -6                                     # Generate the <hashed-password>
      $ echo 'pi:<hashed-password>' | sudo tee userconf.txt   # Put them inside `userconf.txt`

   5. Finally, unmount the image:

      $ sudo umount /mnt/rpi

5. Run qemu emulator:

   $ cd /home/mydir
   $ qemu-system-aarch64 -machine virt -cpu cortex-a72 -smp 6 -m 4G \
       -kernel Image -append "root=/dev/vda2 rootfstype=ext4 rw panic=0 console=ttyAMA0" \
       -drive format=raw,file=2023-05-03-raspios-bullseye-arm64.img,if=none,id=hd0,cache=writeback \
       -device virtio-blk,drive=hd0,bootindex=0 \
       -netdev user,id=mynet,hostfwd=tcp::2222-:22 \
       -device virtio-net-pci,netdev=mynet \
       -monitor telnet:127.0.0.1:5555,server,nowait

   This machine will be able to access the internet.
6. After the machine is completely booted up, you can login to it from your computer by using ssh and the username and password you specified:

   $ ssh -l pi localhost -p 2222

7. Done!

Troubleshooting

• If you had any problem with connecting to internet, it might be because of bad DNS configurations, and you should consider adding nameserver 8.8.8.8 to top of the file /etc/resolv.conf in the machine.
• You can access the qemu monitor console with:

  $ telnet localhost 5555

i got a error with user and passwd; pi and raspberry.

Hey, thanks for trying out the guide!

Did you make sure to put the hashed password value inside userconf.txt? You can generate the hashed password as I mentioned above, but be careful to avoid any typos:

$ openssl passwd -6  # enter your password here, for example 'raspberry'
$6$sbMZ.NX0yYjCn3JL$Wrdd9WKZfXlmpoCgS6O7JEibneWvkHlJwwtENCTssEQ6qt0pvbIciYdOUEortfyyx3WkQUVeXaHPc/ojFGxWv/

Now, use this hashed value like so:

$ echo 'pi:<hashed-password>' | sudo tee userconf.txt

# If you used `raspberry` as password
$ echo 'pi:$6$sbMZ.NX0yYjCn3JL$Wrdd9WKZfXlmpoCgS6O7JEibneWvkHlJwwtENCTssEQ6qt0pvbIciYdOUEortfyyx3WkQUVeXaHPc/ojFGxWv/' | sudo tee userconf.txt

Let me know if the issue continues, and provide any additional details if possible!

Worked like a charm. Thanks for this. Kudos

This is a really useful start, thank you.

I know "real" raspberry pi installs will expand the storage to something usable (e.g., expand the / filesystem to the size of the SD card). What can we do about this with this approach? Any thoughts on how can we get a more usable root disk size with this approach?

EDIT: Never mind, I figured out how to do it:
EDIT 2: Removing this comment in favor of @HarryCutts better method

I found this guide really useful, thank you! I had to also sudo apt install flex bison libssl-dev to build the kernel; might be worth adding them to step 2.

@mmguero: For resizing the partition, I found that I could use virt-resize from the guestfs-tools package to expand the root partition from outside the VM, which avoids having to make partition changes on a running system:

$ cp original.img bigger.img
$ truncate -s +40G bigger.img
$ virt-resize --expand /dev/sda2 original.img bigger.img

Ah, very cool, thank you!

One thing I haven't been able to figure out, so maybe somebody watching this might know: I can see in the built kernel directory the .ko kernel module files that have been built, and I have even played around with trying to share this folder under /lib/modules, but I can't get modprobe foobar (whatever the kernel module is) to work to load any kernel modules in the emulated guest. Specifically I was trying to set up docker in the guest but was unable to load the overlay module.

Nice post! I can even run RPI under wsl. But qemu need -nographic to avoid connecting graphic server.

for wsl2, the command to run qemu should be:
qemu-system-aarch64 -machine virt -cpu cortex-a72 -smp 6 -m 4G -kernel Image -append "root=/dev/vda2 rootfstype=ext4 rw panic=0 console=ttyAMA0" -drive format=raw,file=2020-08-20-raspios-buster-arm64-lite.img,if=none,id=hd0,cache=writeback -device virtio-blk,drive=hd0,bootindex=0 -netdev user,id=mynet,hostfwd=tcp::2222-:22 -device virtio-net-pci,netdev=mynet -monitor telnet:127.0.0.1:5555,server,nowait -nographic
you should also install the package like @HarryCutts mentioned:
sudo apt install flex bison libssl-dev

i got a error with user and passwd; pi and raspberry.

Hey, thanks for trying out the guide!

Did you make sure to put the hashed password value inside userconf.txt? You can generate the hashed password as I mentioned above, but be careful to avoid any typos:

$ openssl passwd -6  # enter your password here, for example 'raspberry'
$6$sbMZ.NX0yYjCn3JL$Wrdd9WKZfXlmpoCgS6O7JEibneWvkHlJwwtENCTssEQ6qt0pvbIciYdOUEortfyyx3WkQUVeXaHPc/ojFGxWv/

Now, use this hashed value like so:

$ echo 'pi:<hashed-password>' | sudo tee userconf.txt

# If you used `raspberry` as password
$ echo 'pi:$6$sbMZ.NX0yYjCn3JL$Wrdd9WKZfXlmpoCgS6O7JEibneWvkHlJwwtENCTssEQ6qt0pvbIciYdOUEortfyyx3WkQUVeXaHPc/ojFGxWv/' | sudo tee userconf.txt

Let me know if the issue continues, and provide any additional details if possible!

i'm sorry I am trying to do this, i set the password to raspberry using this last command you wrote here but still i cant login with

• user: pi
• passw: raspberry

am I missing something? I cant login neither from the console nor the ssh, in ssh it always says 'Permission denied, please try again.', in console it says 'login incorrect' thanks in advance

Solution

i mounted the root partition of the rpi like this

sudo mount -o loop,offset=272629760 2023-05-03-raspios-bullseye-arm64.img /mnt/rpi

then i modified the file /etc/shadow and replaced the hashed password manually, then i saved the changes and unmounted the partition

sudo nano etc/shadow
# find use and replace hashed password, then save and exit
cd 
sudo umount /mnt/rpi

then I started the vm and enjoyed it

Mounting a partition within an image is less error prone by using kpartx -av raspi.img and then linking the loop devices like this:
ln -s /dev/mapper/loop7p2 raspi.img2

don't forget to remove the loop devices using kpartx -dv raspi.img before deleting the image, else they will stick around till reboot.

you may increase the partition size (before creating the loop devices) by simply issuing truncate -s 4G raspi.img this will set the file size to 4 gig (sparse).

I'm trying to use physical microSD from my RPi for debugging. At the moment I have an issue passing physical microSD to qemu using these additional arguments -drive file=/dev/disk13,if=none,format=raw and -append "root=/dev/mmcblk0p7 rootfstype=ext4"
disk13 is the physical microSD, /dev/mmcblk0p7 is root partition for RaspberryOS, qemu boots the kernel, but then I receive Kernel Panic, because it can't find root partition:

[    2.715128] VFS: Cannot open root device "mmcblk0p7" or unknown-block(0,0): error -6
[    2.715856] Please append a correct "root=" boot option; here are the available partitions:
[    2.716698] 0100            4096 ram0 
[    2.716746]  (driver?)
[    2.716996] 0101            4096 ram1 
[    2.717016]  (driver?)
[    2.718510] 0102            4096 ram2 
[    2.718532]  (driver?)
[    2.718744] 0103            4096 ram3 
[    2.718762]  (driver?)
[    2.719030] 0104            4096 ram4 
[    2.719048]  (driver?)
[    2.719880] 0105            4096 ram5 
[    2.719918]  (driver?)
[    2.720546] 0106            4096 ram6 
[    2.720562]  (driver?)
[    2.720752] 0107            4096 ram7 
[    2.720768]  (driver?)
[    2.720956] 0108            4096 ram8 
[    2.720972]  (driver?)
[    2.722830] 0109            4096 ram9 
[    2.722852]  (driver?)
[    2.723214] 010a            4096 ram10 
[    2.723294]  (driver?)
[    2.724030] 010b            4096 ram11 
[    2.724056]  (driver?)
[    2.724530] 010c            4096 ram12 
[    2.724550]  (driver?)
[    2.724744] 010d            4096 ram13 
[    2.724762]  (driver?)
[    2.724968] 010e            4096 ram14 
[    2.725448]  (driver?)
[    2.725838] 010f            4096 ram15 
[    2.725856]  (driver?)
[    2.726386] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I've tried -hda for qemu, different variants in -append "root=/dev/sda2" etc. Out of ideas what can be wrong - not a frequent qemu user for such scenarios. Qemu docs haven't helped, also for googling - everybody uses images, but not physical microSD.

I'm following the instructions as provided and, while running Step 3 (ARCH=arm64 CROSS_COMPILE=/bin/aarch64-linux-gnu- make -j8) I get the following error:

  DTC     arch/arm64/boot/dts/qcom/sm8150-sony-xperia-kumano-griffin.dtb                                               
certs/extract-cert.c:21:10: fatal error: openssl/bio.h: No such file or directory                                      
   21 | #include <openssl/bio.h>                                                                                       
      |          ^~~~~~~~~~~~~~~                                                                                       
compilation terminated.                                    
make[2]: *** [scripts/Makefile.host:114: certs/extract-cert] Error 1                                                   
make[1]: *** [scripts/Makefile.build:503: certs] Error 2                                                               
make[1]: *** Waiting for unfinished jobs....                                                                           

This has the whole make process exist with an error:
make: *** [Makefile:2010: .] Error 2

If I re-run the command, I get a similar error:

  CC      fs/notify/dnotify/dnotify.o                      
certs/extract-cert.c:21:10: fatal error: openssl/bio.h: No such file or directory                                      
   21 | #include <openssl/bio.h>     
      |          ^~~~~~~~~~~~~~~ 
compilation terminated.                   
make[2]: *** [scripts/Makefile.host:114: certs/extract-cert] Error 1                                                   
make[1]: *** [scripts/Makefile.build:503: certs] Error 2
make[1]: *** Waiting for unfinished jobs....               

certs/extract-cert.c:21:10: fatal error: openssl/bio.h: No such file or directory

did you install - well - openssl-dev package or something like that?
Which platform are you on?
Usually each major distribution got an online package search that allows finding files in packages.

Sorry for polluting this page with my issues. I'm fumbling in the dark here, being rather new to cross compiling and a total Qemu noob. I can only ask really silly questions.

I have OpenSSL installed for dev purposes but alas, it's for my host: x86_64; I suppose I need to download the OpenSSL source and build that for arm64 then figure out how to work it into the steps outlined at the very top of this page. A quick look at the OpenSSL build instructions is making this seem more daunting then expected.

All I want to do is build some "generic" software for the RPiZ2W, RPi3B+, and RPi4B - all aarch64/arm64 ... and by "generic" I mean I'm not making use of any special hardware or capabilities present on the Pi. I've had minimal success with straight-up cross compiling - "Hello World" is easy - but you throw 200+ source files with fancy makefiles and custom libraries at it and things fall down quickly. I thought emulating the RPi OS and migrating my build environment would be easier. It's turning out not to be.

In the past, I would have dedicated some actual hardware to be the "build machine" but instead of going out and buying yet another RPi4 just for building, I thought I'd be smarter about it and cross-compile or emulate.

Source (and dev-package) is per se platform independent. The dev package is the one that contains the header files .h and this is what you need to compile a software. You do not need the source. It asked for the .h file, remember?

again: which system do you use for building? then I can point you to which packages you need to install.

@LiamFry,

You simply need to install the libssl-dev package:

sudo apt-get install libssl-dev

I cannot get the network to work on my M2, everything else seems perfect.

How to run armhf version?

Thank you so much for you work!
Based on it, I've figured out (through a lot of work!) a way to run a Raspberry Pi OS Desktop VM with a working interactive GUI!
See here, I've got a fully automated script for it:
https://gist.github.com/emendir/922d6914a1705ed2e8e4e96db726c422

I'm trying to use physical microSD from my RPi for debugging. At the moment I have an issue passing physical microSD to qemu using these additional arguments -drive file=/dev/disk13,if=none,format=raw and -append "root=/dev/mmcblk0p7 rootfstype=ext4" disk13 is the physical microSD, /dev/mmcblk0p7 is root partition for RaspberryOS, qemu boots the kernel, but then I receive Kernel Panic, because it can't find root partition:

[    2.715128] VFS: Cannot open root device "mmcblk0p7" or unknown-block(0,0): error -6
[    2.715856] Please append a correct "root=" boot option; here are the available partitions:
[    2.716698] 0100            4096 ram0 
[    2.716746]  (driver?)
[    2.716996] 0101            4096 ram1 
[    2.717016]  (driver?)
[    2.718510] 0102            4096 ram2 
[    2.718532]  (driver?)
[    2.718744] 0103            4096 ram3 
[    2.718762]  (driver?)
[    2.719030] 0104            4096 ram4 
[    2.719048]  (driver?)
[    2.719880] 0105            4096 ram5 
[    2.719918]  (driver?)
[    2.720546] 0106            4096 ram6 
[    2.720562]  (driver?)
[    2.720752] 0107            4096 ram7 
[    2.720768]  (driver?)
[    2.720956] 0108            4096 ram8 
[    2.720972]  (driver?)
[    2.722830] 0109            4096 ram9 
[    2.722852]  (driver?)
[    2.723214] 010a            4096 ram10 
[    2.723294]  (driver?)
[    2.724030] 010b            4096 ram11 
[    2.724056]  (driver?)
[    2.724530] 010c            4096 ram12 
[    2.724550]  (driver?)
[    2.724744] 010d            4096 ram13 
[    2.724762]  (driver?)
[    2.724968] 010e            4096 ram14 
[    2.725448]  (driver?)
[    2.725838] 010f            4096 ram15 
[    2.725856]  (driver?)
[    2.726386] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I've tried -hda for qemu, different variants in -append "root=/dev/sda2" etc. Out of ideas what can be wrong - not a frequent qemu user for such scenarios. Qemu docs haven't helped, also for googling - everybody uses images, but not physical microSD.

Did you fixed it? I am writing because I want to emulate a Yocto generated Raspberry Pi Image in Qemu.

This is fantastic! I have been writing a raspberry emulator in a docker image for a couple years now. Originally I used the raspi3b board, combined with the 5.4-y kernel branch. When upgrading to 6.1-y, the setup broke due some issue with ttyAMA0 being moved to ttyAMA1 on the raspi3b (long story). For some reason I could not get to the login prompt.

It took me a full day of googling to stumble on this gist. I noticed you were using 6.1.34. And I can confirm that it work! I've even got the raspberry image to boot with 6.6-y kernel and the 2024-07-04-raspios-bookworm-arm64-lite.img image!

The only thing that I had to do (which is the issue that @superseppl ) ran into, is to map vda, vda1 and vda2 to mmcblk0, mmcblk0p1 and mmcblk0p2. This was possible by editing the image and adding a udev rule that symlinks the virtual hard drive as a virtual sd card.

For the interested, you can find the complete setup in my pi-ci repo

Thanks so much!

[ 2.725856] (driver?)
[ 2.726386] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I've tried -hda for qemu, different variants in -append "root=/dev/sda2" etc. Out of ideas what can be wrong - not a frequent qemu user for such scenarios. Qemu docs haven't helped, also for googling - everybody uses images, but not physical microSD.

Did you fixed it? I am writing because I want to emulate a Yocto generated Raspberry Pi Image in Qemu.

I was trying to do a similar thing and I was able to solve the error related to rootfs increasing the bootdelay from kernel cmdline (using the -sd argument with qemu "machine rasp4"). But even with that, systemd from rasp os gets stucked.

Emulating raspi3b is a bit easier than the generic virt because we can simply extract the kernel8.8.img from the raspberry image, without having to compile the linux kernel manually.

qemu-system-aarch64 \
    -machine raspi3b \
    -cpu cortex-a72 \
    -m 1G  \
    -smp 4 \
    -sd ${IMAGE_FILE} \
    -kernel ${KERNEL} \
    -dtb ${DEVICE_TREE_BLOB} \
    -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootdelay=1" \
    -netdev user,id=net0,hostfwd=tcp::2264-:22 \
    -device usb-net,netdev=net0 \
    -device usb-mouse -device usb-kbd \
    -msg timestamp=on

The above command will also give you graphics if:

• you use the latest qemu, or at least above v6.2.0, which is what I am getting from apt on Debian.
  That version has a bug which prevents it from creating /dev/fb0 framebuffer, and without that bad boy you have no graphics.
  I went on a wild goose chase trying to compile the kernel with CONFIG_DRM_FBDEV_EMULATION=y and similar params, but that wasn't the problem; the defaults are ok.

With qemu v6.2.0 we see this error during boot:

pi@raspberrypi:/dev$ dmesg | grep -i fb
[    1.501182] bcm2708_fb soc:fb: Unable to determine number of FBs. Disabling driver.
[    1.501286] bcm2708_fb: probe of soc:fb failed with error -2

• grab raspios buster.
  2021-05-07-raspios-buster-arm64.img - graphics work
  2024-07-04-raspios-bullseye-arm64.img - the logo appears, but after that the graphics don't. don't know why
  2024-07-04-raspios-bookworm-arm64.img - bookworm is even more tricky to get to work than bullseye, even the console won't work

Both of these points are also mentioned here.

If you want to use -machine virt, the advantage is you can give it more cores and ram. But you will need to find the right flags to attach graphics, because otherwise you will get just the console, and you'll see that the /dev/fb0 is not there.

Works great for me but I have a question. I was initially trying to use this guide and precompiled kernel image and that guide uses a .dtb file and you do not. What's the difference / advantage / disadvantage?

Hey @TheBrick2, think about it like this.
The raspberry .img that we download is not designed for qemu specifically (it's designed for the physical Raspberry),
so its kernel (the kernel8.img inside it) is probably not compiled with kvm_guest config, it's more "generic".

However, '-machine raspi3b' flag (which I used above) works around this, because even though the kernel is generic, it knows the specific machine type: raspi3b.

But when the machine type is also generic (e.g. '-machine virt', or -M versatilepb, used by the guide you linked) it doesn't work, and we need the kernel to be specific.

So that's why in that case, we need to compile the kernel with kvm_guest, or get it pre-compiled from somewhere, e.g. that repo you mentiond.

PS: new account, but I'm the guy who posted the message you quoted.

Thank you a lot, dear friend!!

I got it working under the Apple Hypervisor with virtio graphics. The M4 is fast enough to run llvmpipe. Benchmark: https://browser.geekbench.com/v6/cpu/11278261

https://gist.github.com/jamesy0ung/b411143c15d3c8296684c4987e7c9894