Measuring Firecracker boot time

Firecracker comes with an internal way of logging a timestamp that measures time elapsed between the very start of the guest VM and the moment a specific IO port has been written to.

That allows for marking specific moment along the boot process by having code writing to this port.

Artifacts build

Here we're going to measure the time it takes for a Firecracker guest VM to reach userspace. To do so we're going to build 3 components:

Firecracker
A guest kernel
A rootfs image.

We assume you created a dedicated build directory and export it to FC_BUILD. For example

export FC_BUILD=~/fc-build
mkdir -p $FC_BUILD

Firecracker

pushd $FC_BUILD
git clone https://github.com/firecracker-microvm/firecracker.git
cd firecracker
./tools/devtool/build
popd $FC_BUILD

The firecracker binary is going to be at $FC_BUILD/firecracker/build/debug/firecracker

Guest Kernel

pushd $FC_BUILD
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-stable
cd linux-stable
git reset --hard v4.14.72
curl https://raw.githubusercontent.com/firecracker-microvm/firecracker/master/resources/microvm-kernel-config -o .config
make oldconfig
make vmlinux -j `nproc`
popd $FC_BUILD

The Firecracker guest kernel is going to be at $FC_BUILD/linux-stable/vmlinux

Rootfs

See Firecracker rootfs preparation instructions

First we create an empty ext4 rootfs image:

export ROOTFS_DIR=$FC_BUILD/rootfs
mkdir -p $ROOTFS_DIR
dd if=/dev/zero of=$ROOTFS_DIR/fc-rootfs.ext4 bs=1M count=50
mkfs.ext4 $ROOTFS_DIR/fc-rootfs.ext4
mkdir /tmp/fc-rootfs
sudo mount $ROOTFS_DIR/fc-rootfs.ext4 /tmp/fc-rootfs

We can now populate it with an Alpine Linux image:

sudo docker run -it --rm -v /tmp/fc-rootfs:/fc-rootfs alpine

# You are now inside the container.
# First we will add an init system and some utilities
apk add openrc
apk add util-linux

# Set up a login terminal on the serial console (ttyS0):
ln -s agetty /etc/init.d/agetty.ttyS0
echo ttyS0 > /etc/securetty
rc-update add agetty.ttyS0 default

# Make sure special file systems are mounted on boot:
rc-update add devfs boot
rc-update add procfs boot

# Then, copy the newly configured system to the rootfs image:
for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C /fc-rootfs; done
for dir in dev proc run sys var; do mkdir /fc-rootfs/${dir}; done

# All done, exit docker shell
exit

Now we want to add a small utility that poke the Firecracker debug io port as soon as we reach userspace.

We first need to build a small piece of code to do so:

pushd $FC_BUILD
git clone https://github.com/sameo/firecracker-magic-port.git
cd firecracker-magic-port/
make
popd

Then we copy this binary into our rootfs:

sudo cp fc-magic-port /tmp/fc-rootfs/sbin/

The final step is to create a custom init command to have it first calling into fc-magic-port and then starting the regular init:

sudo bash
cat <<EOM >/tmp/fc-rootfs/init
#!/bin/sh
/sbin/fc-magic-port
exec /sbin/init
EOM
chmod a+x /tmp/fc-rootfs/init
exit

We can now unmount the rootfs

sudo umount /tmp/fc-rootfs

The Firecracker guest rootfs is now at $FC_BUILD/rootfs/fc-rootfs.ext4

Artifacts summary

Artifact	Location
Firecracker VMM	$FC_BUILD/firecracker/build/debug/firecracker
Guest kernel	$FC_BUILD/linux-stable/vmlinux
Guest rootfs	$FC_BUILD/rootfs/fc-rootfs.ext4

Running and measuring

Start a Firecracker instance:

rm -rf /tmp/firecracker.socket && $FC_BUILD/firecracker/build/debug/firecracker --seccomp-level 0

On another terminal, start a Firecracker guest:

$FC_BUILD/firecracker-magic-port/start_instance-debug.sh /tmp/firecracker.socket

Then the guest boot time timestamp can be found in the Firecracker log pipe:

grep Guest-boot-time /tmp/logs.fifo
2019-05-30T12:15:18.698648314 [anonymous-instance] Guest-boot-time = 279446 us 279 ms, 272485 CPU us 272 CPU ms

sameo/firecracker-boot-time.md