Running virtualized x86_64 and emulated arm64 Ubuntu cloud images using QEMU

arm64.md

QEMU arm64 cloud server emulation

This is basically a rehash of an original post on CNXSoft - all credit (particularly for the Virtio device arguments used below) belongs to the author of that piece.

Download the latest uefi1.img image. E.g. ubuntu-16.04-server-cloudimg-arm64-uefi1.img from https://cloud-images.ubuntu.com/releases/16.04/release/

Download the UEFI firmware image QEMU_EFI.fd from https://releases.linaro.org/components/kernel/uefi-linaro/latest/release/qemu64/

Determine your current username and get your current ssh public key:

$ whoami
ghawkins
$ cat ~/.ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC...

Use these values to create a cloud.txt file replacing the username, here shown as ghawkins, and the ssh-rsa value with the values appropriate for you:

$ cat > cloud.txt << EOF
#cloud-config
users:
  - name: ghawkins
    ssh-authorized-keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC...
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: sudo
    shell: /bin/bash
EOF

Important: the #cloud-config line above is not a comment and things will fail silently without it.

Cread a cloud-config disk image:

$ cloud-localds --disk-format qcow2 cloud.img cloud.txt

Note: by default cloud-localds creates a raw image and QEMU now complains at having to guess about such an image so use --disk-format qcow2 to specify a well defined format that QEMU can easily consume.

Backup your image:

$ cp ubuntu-16.04-server-cloudimg-arm64-disk1.img ubuntu-16.04-server-cloudimg-arm64-disk1.img.orig

The QEMU launch command is somewhat more complex than for e.g. a fully virtualized, rather than emulated, setup with an x86_64 guest running on an x86_64 host.

Here is the command first:

$ qemu-system-aarch64 \
    -smp 2 \
    -m 1024 \
    -M virt \
    -cpu cortex-a57 \
    -bios QEMU_EFI.fd \
    -nographic \
    -device virtio-blk-device,drive=image \
    -drive if=none,id=image,file=ubuntu-16.04-server-cloudimg-arm64-uefi1.img \
    -device virtio-blk-device,drive=cloud \
    -drive if=none,id=cloud,file=cloud.img \
    -device virtio-net-device,netdev=user0 \
    -netdev user,id=user0 \
    -redir tcp:2222::22

You'll have to change ubuntu-16.04-server-cloudimg-arm64-uefi1.img if you downloaded a later image with a different name.

Now let's look at the arguments that configure our system:

• -smp 2 - 2 (virtual) cores.
• -m 1024 - 1024MB of system memory.
• -M virt - emulate a generic QEMU ARM machine.
• -cpu cortex-a57 - the CPU model to emulate.
• -bios QEMU_EFI.fd - the BIOS firmware file to use.
• -nographic - output goes to the terminal (rather than opening a graphics capable window).
• -device virtio-blk-device,drive=image - create a Virtio block device called "image".
• -drive if=none,id=image,file=ubuntu-16.04-server-cloudimg-arm64-uefi1.img - create a drive using the "image" device and our cloud server disk image.
• -device virtio-blk-device,drive=cloud - create another Virtio block device called "cloud".
• -drive if=none,id=cloud,file=cloud.img - create a drive using the "cloud" device and our cloud-config disk image.
• -device virtio-net-device,netdev=user0 - create a Virtio network device called "user0"
• -netdev user,id=user0 - create a user mode network stack using device "user0"
• -redir tcp:2222::22 - map port 2222 on the host to port 22 (the standard ssh port) on the guest.

Here we create a generic QEMU ARM machine. You can see a complete list of possible ARM machines like so:

$ qemu-system-aarch64 -M help
akita                Sharp SL-C1000 (Akita) PDA (PXA270)
...
z2                   Zipit Z2 (PXA27x)

This list seems to include all ARM machines, not just 64-bit ones. The latest versions of QEMU (but not the one that currently comes with Ubuntu 16.04 LTS) include the well know Raspberry Pi 2 (but not the 3).

For a given machine you can then see the supported processors:

$ qemu-system-aarch64 -M virt -cpu help
 arm1026
 ...
 ti925t

Once you run the command up above to launch an emulated ARM64 machine it will take a few minutes to boot and will output something like the following:

error: no suitable video mode found.
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset

The initial error, about "no suitable video mode found", can be ignored - we specifically set -nographic.

Eventually a login prompt will appear - which cannot be used as in our cloud-config file we only specified key based ssh login.

Depending on how fast various jobs (kicked off during the boot process) run further output will appear after the login prompt appears.

The first time you launch a given system you should see output confirming that the ssh key specified up above has been installed.

And eventually you should see something like:

[  220.784509] cloud-init[1358]: Cloud-init v. 0.7.8 finished at ...

Now in another terminal you can log in to the newly launched cloud server:

$ ssh -p 2222 ghawkins@localhost

If all goes well you'll log straight in without any username or password.

If you've started previous QEMU images in a similar manner then ssh may issue a dire warning like so (and refuse to login):

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

To resolve this and remove previous details:

$ ssh-keygen -f ~/.ssh/known_hosts -R '[localhost]:2222'

When logged into the cloud server you can...

• Confirm that it's an aarch64 system:

$ uname -a
Linux ubuntu 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:37:14 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

• Has two cores:

$ cat /proc/cpuinfo
processor   : 0
...

processor   : 1
...

• Shut it down:

$ sudo shutdown now

In the original terminal (where you launched qemu-system-aarch64) you can follow the shutdown process.

Note: when running sudo shutdown now the shutdown succeeds but the following error appears:

sudo: unable to resolve host ubuntu

You'll see this anytime you run sudo - to resolve it (as per Ask Ubuntu) just edit /etc/hosts and add ubuntu at the end of the existing line for the address 127.0.0.1 so you end up with something like:

127.0.0.1 localhost ubuntu

ubuntu-cloud-image-with-qemu.md

QEMU x86_64 cloud server virtualization

Get a cloud image from:

https://cloud-images.ubuntu.com/releases/

E.g. https://cloud-images.ubuntu.com/releases/16.10/release/ubuntu-16.10-server-cloudimg-amd64.img

Create a cloud-config called cloud.txt, which defines who can login etc. to the virtual cloud server, and create a disk image from it. For this you need your login name on your current system, along with the public part of your current ssh key:

$ whoami
ghawkins
$ cat ~/.ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC...
$ cat > cloud.txt << EOF
#cloud-config
users:
  - name: ghawkins
    ssh-authorized-keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC...
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: sudo
    shell: /bin/bash
EOF
$ cloud-localds cloud.img cloud.txt

Copy the line contained in id_rsa.pub into the ssh-authorized-keys section and replace the username specified by name with your username.

Important: I thought #cloud-config was a comment and left it out - but without it no error is reported but you cannot login later.

Backup your image:

$ cp ubuntu-16.10-server-cloudimg-amd64.img ubuntu-16.10-server-cloudimg-amd64.img.orig

Note: this is a compressed qcow2 image - while it's about 320MB the running guest will see it as 2GB (as we'll confirm later).

Now start the cloud guest:

$ qemu-system-x86_64 \
    -enable-kvm \
    -smp 2 \
    -m 1024 \
    -nographic \
    -hda ubuntu-16.10-server-cloudimg-amd64.img \
    -hdb cloud.img \
    -redir tcp:2222::22

The command line arguments:

• -enable-kvm - full virtualization (rather than emulation).
• -smp 2 - two (virtual) processors (as we'll confirm later).
• -m 1024 - 1024MB of system memory.
• -nographic - output goes to the terminal (rather than opening a graphics capable window).
• -hda ubuntu-16.10-server-cloudimg-amd64.img - use our Ubuntu cloud image as the primary drive.
• -hdb cloud.img - use the image we created from cloud.txt as the secondary drive.
• -redir tcp:2222::22 - map port 2222 on the host to port 22 (the standard ssh port) on the guest.

Once booted you eventually get to the console getty login prompt. No one can login here - so you need to switch to another terminal tab.

Now let's logon to the guest using the redirected port and check out a few things and then shut down the guest:

$ ssh -p 2222 ghawkins@localhost
Welcome to Ubuntu 16.10 (GNU/Linux 4.8.0-26-generic x86_64)
...
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            491M     0  491M   0% /dev
tmpfs           100M  3.2M   97M   4% /run
/dev/sda1       2.0G  979M 1016M  50% /
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
/dev/sda15      105M  4.8M  100M   5% /boot/efi
tmpfs           100M     0  100M   0% /run/user/1000

$ cat /proc/cpuinfo 
processor   : 0
vendor_id   : GenuineIntel
...

processor   : 1
vendor_id   : GenuineIntel
$ sudo shutdown now
Connection to localhost closed by remote host.

So above using df -h we can see that the disk appears to be 2GB and with cat /proc/cpuinfo we can see that we appear to have two processors. Finally using shutdown we can get back to the command prompt in the terminal where the guest was started.

TODO: see how changing the number of virtual CPUs affects the performance of the guest.

If you redo everything from scratch again with a copy of the original disk image then the guest will generate a new key to identify itself which will cause ssh to refuse to allow you to reconnect due to the change in key. To remove the old key from known_hosts do:

$ ssh-keygen -f ~/.ssh/known_hosts -R '[localhost]:2222'

Working out how to get this far was down to:

• http://www.cnx-software.com/2016/05/10/how-to-run-ubuntu-16-04-aarch64-64-bit-arm-cloud-images-on-your-intelamd-linux-computer/ - yes, even though it's about running an emulated ARM guest it was helpful in working out how to get everything going.
• https://www.digitalocean.com/community/tutorials/how-to-use-cloud-config-for-your-initial-server-setup - goes into the cloud config in more detail (and convinced me to finally add #cloud-config at the start of the file).

The Ubuntu cloud images page wasn't as helpful as it should be:

• https://help.ubuntu.com/community/UEC/Images

But it does cover uncompressing the qcow2 disk image and increasing its size (2GB isn't much) and fancier stuff like creating a delta image to keep your initial disk image in a pristine condition.

Also shown at: https://askubuntu.com/questions/281763/is-there-any-prebuilt-qemu-ubuntu-image32bit-online/1081171#1081171

Thanks for the very useful gits, I'm trying to run the same image using libvirt any ideas on how to write the xml file?
I'm stuck with some error in passing the bios file

Hi, I followed this, but qemu gives me "-redir : invalid option". Without -redir option, it goes go login prompt, but I can't login in using ssh of course. What can I do? my qemu-system-aarch64 version is 5.1.0.

@chankim - I think this SO answer covers your issue.

Hi, George,
Thank you for this good information. I applied your SO answer(-nic user,hostfwd=tcp::5022-:22 instead of -redir ..).
This time it took much longer to the login prompt.
And near the end I saw this message below. (Some key values are modified for this post). I'm not sure if this was ok. (seems ok)
And I tried 'ssh -p 2222 ckim@localhost' but access was denied. I also tried ssh -p 5022 ckim@localhost in vain.
I would appreciate it you can give me any suggestion (or anyone else?) . Thanks!
(I don't know why many lines below are stroke-out)

Ubuntu 16.04.7 LTS ubuntu ttyAMA0

ubuntu login: [ 132.546717] cloud-init[1239]: Generating locales (this might take a while)...
[ 136.461912] cloud-init[1239]: en_US.UTF-8... done
[ 136.469791] cloud-init[1239]: Generation complete.
[ 139.689257] cloud-init[1239]: Cloud-init v. 20.4-0ubuntu116.04.1 running 'modules:config' at Thu, 14 Jan 2021 08:44:54 +0000. Up 131.08 seconds.
ci-info: Authorized keys from /home/ckim/.ssh/authorized_keys for user ckim
ci-info: +---------+----------------------+---------+---------+
ci-info: | Keytype | Fingerprint (sha256) | Options | Comment |
ci-info: +---------+----------------------+---------+---------+
ci-info: | ssh-rsa | ? | - | - |
ci-info: +---------+----------------------+---------+---------+
<14>Jan 14 08:45:06 ec2:
<14>Jan 14 08:45:06 ec2: #############################################################
<14>Jan 14 08:45:06 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Jan 14 08:45:06 ec2: 1024 SHA256:mFHGCCcfGk+nnlJLpTfTuOP7ydqwTS4bxn/GiR2+F7s root@ubuntu (DSA)
<14>Jan 14 08:45:06 ec2: 256 SHA256:H1XuP8WyUffzDE8tLam168jbNECxav0bhVSMsBmxzDs root@ubuntu (ECDSA)
<14>Jan 14 08:45:06 ec2: 256 SHA256:53+YF/q6aN7z69mFjhXDptxBo1b89/2gU3bgigHY234 root@ubuntu (ED25519)
<14>Jan 14 08:45:06 ec2: 2048 SHA256:0Dv9EKcmMIJ9sqBgxTwBMbFcP3YPduK6Nbj55lnPqFk root@ubuntu (RSA)
<14>Jan 14 08:45:06 ec2: -----END SSH HOST KEY FINGERPRINTS-----
<14>Jan 14 08:45:06 ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHXyNTYAAABBBGxXZFaUcE32JooNrxw2LkQYDxEFpblTABtSgfY3R8DYpasGreD6CQFP6L5xYk1h/EETL+08kwprOIWIUS07ftg= root@ubuntu
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGICPN2DsGch1AW+1MilQzN+yYMypAmBt71bEii03pX7 root@ubuntu
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABBAABAQDLZIy7XpwYTNjYNbLc3xgK/9rGPEoLNwQH0ETLtQuiUYp/Oy9+TblXrQu8XsEJ3AhdpGePdMs7OKExf5LJnh4F+2HZO3P7WkNYKOPvhwyF0xl7UEABuy1FmUpEo7qvdjA7kE3ez9ymF+ix1DOqlH3Wo2gol+JISkQfeJOAiawBtrTw/tl2LqKh7wRe78bZJ950vpc7UKliAGdvES+KKTJW+rds3+bVb9nHx8hZk4yR0+IP8nWTeCOS5lc4kcf2PxNDoAK/kGJ8iXBM8Kt9i9j9WYEyMAoRNxiCbFLhDUGKoWhFQLnlk0qC4Ltei35laN2yD7jIMn/vWn2SsAvesNcR root@ubuntu
-----END SSH HOST KEY KEYS-----
[ 143.845149] cloud-init[1283]: Cloud-init v. 20.4-0ubuntu116.04.1 running 'modules:final' at Thu, 14 Jan 2021 08:45:05 +0000. Up 142.20 seconds.
[ 143.858762] cloud-init[1283]: Cloud-init v. 20.4-0ubuntu1~16.04.1 finished at Thu, 14 Jan 2021 08:45:07 +0000. Datasource DataSourceNoCloud [seed=/dev/vda][dsmode=net]. Up 143.75 seconds

How to launch above scripts with arm-trusted-firmware or optee?