Booting Flatcar on firecracker

Booting-Flatcar-on-firecracker.md

Step-by-step instructions to boot Flatcar on Firecracker.

Get Firecracker and firectl

Check for the latest versions on https://github.com/firecracker-microvm/firecracker/releases and https://github.com/firecracker-microvm/firectl/releases ; for our example we'll go with 0.24.6 and 0.1.0.

wget https://github.com/firecracker-microvm/firecracker/releases/download/v0.24.6/firecracker-v0.24.6-x86_64.tgz
wget https://firectl-release.s3.amazonaws.com/firectl-v0.1.0
tar xzf firecracker-v0.24.6-x86_64.tgz
mv release-*/* .
chmod 755 firecracker-* firectl-*

Get Flatcar OS image and kernel

We'll need the uncompressed vmlinux image for use with firecracker which we can obtain from a release's vmlinuz using the extract-vmlinux script. Note that auto-detection of the compression algorithm can fail (Flatcar uses zstd), so it is recommended to apply the below patch to the script before running it.

--- extract-vmlinux	2021-11-05 14:28:47.123984433 +0100
+++ extract-vmlinux 2021-11-05 14:28:29.487982001 +0100
@@ -49,12 +49,17 @@
 trap "rm -f $tmp" 0
 
 # That didn't work, so retry after decompression.
-try_decompress '\037\213\010' xy    gunzip
-try_decompress '\3757zXZ\000' abcde unxz
-try_decompress 'BZh'          xy    bunzip2
-try_decompress '\135\0\0\0'   xxx   unlzma
-try_decompress '\211\114\132' xy    'lzop -d'
-try_decompress '\002!L\030'   xxx   'lz4 -d'
+#try_decompress '\037\213\010' xy    gunzip
+#try_decompress '\3757zXZ\000' abcde unxz
+#try_decompress 'BZh'          xy    bunzip2
+#try_decompress '\135\0\0\0'   xxx   unlzma
+#try_decompress '\211\114\132' xy    'lzop -d'
+#try_decompress '\002!L\030'   xxx   'lz4 -d'
+
+# check if unzstd is installed, fail if it is not
+set -e
+unzstd -V >/dev/null
+set +e
 try_decompress '(\265/\375'   xxx   unzstd
 
 # Finally check for uncompressed images or objects:

For this example I'll go with Alpha release 3033.0.0, but any recent release should work.

wget https://alpha.release.flatcar-linux.net/amd64-usr/3033.0.0/flatcar_production_image.bin.bz2
wget https://alpha.release.flatcar-linux.net/amd64-usr/3033.0.0/flatcar_production_image.vmlinuz
bunzip2 flatcar_production_image.bin.bz2

Now, extract the uncompressed vmlinux kernel from vmlinuz:

./extract-vmlinux flatcar_production_image.vmlinuz >vmlinux

Extract verity hash and figure out partition UUIDs

We'll need both the USR partition's verity hash as well as the ROOT and USR partition's UUIDs as input for firecracker when starting the VM. The verity hash is stored in vmlinuz at a 64byte offset; dd can help us extracting it. Then we'll use partx to extract partition UUIDs.

Verity Hash

dd if=flatcar_production_image.vmlinuz conv=notrunc skip=64 count=64 bs=1 status=none
9b97164672bde72b80b94296964fb69ddca38e99ac815b11a20ca2c3847d23b1

Take note of the verity hash, we'll need it in the final step.

Partition UUIDs

partx -s flatcar_production_image.bin | grep USR-A | tr -s " " | cut -d " " -f 8

yields (for USR-A)

7130c94a-213a-4e5a-8e26-6cce9662f132

and

partx -s flatcar_production_image.bin | grep ROOT | tr -s " " | cut -d " " -f 8

yields (for ROOT)

3be5295f-0f3b-4906-8b72-e8d34102c906

Finally, run Firecracker

The generic command line for running Flatcar on Firecracker is

./firectl-v0.1.0 --firecracker-binary=./firecracker-v0.24.6-x86_64 \
    --kernel=[FLATCAR_BINPKG_VMLINUZ] --root-drive=flatcar_production_image.bin \
    --root-partition=[ROOT_PARTITION_UUID] \
    --kernel-opts="rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=[USR_PARTITION_UUID] consoleblank=0 console=ttyS0,115200n8 console=tty0 flatcar.autologin verity.usrhash=[VERITY_HASH] rd.driver.pre=virtio_mmio console=ttyS0 reboot=k panic=1 i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd"

So for the above example, the concrete command line is

./firectl-v0.1.0 --firecracker-binary=./firecracker-v0.24.6-x86_64 \
    --kernel=vmlinux --root-drive=flatcar_production_image.bin \
    --root-partition=3be5295f-0f3b-4906-8b72-e8d34102c906 \
    --kernel-opts="rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 consoleblank=0 console=ttyS0,115200n8 console=tty0 flatcar.autologin verity.usrhash=9b97164672bde72b80b94296964fb69ddca38e99ac815b11a20ca2c3847d23b1 rd.driver.pre=virtio_mmio console=ttyS0 reboot=k panic=1 i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd"

This will boot Flatcar in firecracker. You'll get a shell (since we use flatcar.autologin) over an emulated serial console. Note that the first boot takes a bit longer than subsequent boots since the initial provisioning set-up runs. Subsequent boots will be faster.

On my test laptop, Flatcar took (from zero to shell) ~6.7 seconds to boot initially, and ~5 seconds for subsequent boots.

This is awesome. Thanks @t-lo. I was able to get this working without having to install the Gnome tools using partx. For example,

verity=`dd if=flatcar_production_image.vmlinuz conv=notrunc skip=64 count=64 bs=1 status=none`
root=`partx -s flatcar_production_image.bin | grep ROOT | tr -s " " | cut -d " " -f 8`
usr=`partx -s flatcar_production_image.bin | grep USR-A | tr -s " " | cut -d " " -f 8`

./firectl-v0.1.0 --firecracker-binary=/usr/local/bin/firecracker \
    --kernel=vmlinux-5.10.75-flatcar --root-drive=flatcar_production_image.bin \
    --root-partition=$root \
    --kernel-opts="rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=$usr consoleblank=0 console=ttyS0,115200n8 console=tty0 flatcar.autologin verity.usrhash=$verity rd.driver.pre=virtio_mmio console=ttyS0 reboot=k panic=1 i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd"

@jeffmcaffer Great feedback, thank you! I'll update the gist accordingly.

@jeffmcaffer ==> https://gist.github.com/t-lo/f6b1d7589ccf9c8b89a83b501a258132

I'll briefly discuss the script in our community call today.