buildroot-qemu.md

introduction

Today I received a notification that TLSe which uses libtomcrypt and libtommath had an issue on Big-Endian MIPS32.

I took this as a reason to finally write down this tutorial on how to build a toolchain, linux kernel and root-fs for a foreign architecture, in this example MIPS32, with Buildroot to run your cross-compiled software on.

We also need QEMU to run all this and I use the system-provided QEMU version in this example. Building QEMU is pretty straight-forward if you follow their examples if you want to build it from source. so I also describe the build process of QEMU for MIPS32.

First let's say the most important, the Buildroot&QEMU documentation is fine and pretty complete and this basically just sums up all the quirks I had to get this up and running.

We use the latest Buildroot version from Git.

buildroot & linux-kernel

initial setup

git clone --depth 1 git://git.buildroot.net/buildroot

To see all available default-configurations provided by Buildroot (no need to do this):

cd buildroot
# list all default-configs
make list-defconfigs
# go back to the previous folder
cd -

Alright, we want to build for MIPS32 so we choose the Buildroot defconfig qemu_mips32r6_malta_defconfig

We're gonna do an out-of-tree build, so we can use one Buildroot repository for several target-configurations.

mkdir buildroot_mips32be
cd buildroot_mips32be/
make O=$PWD -C ../buildroot qemu_mips32r6_malta_defconfig

configuration

We want to run cross-compiled stuff that lies on the host-pc, so we need some components. The easiest way I found to do that with QEMU is to use the VirtIO Plan9 network-filesystem.

buildroot config

We use eudev as it makes our life easier with the automatic setup of devices within the target OS.

Eventually you need a apt install libncurses5-dev to get the menuconfig running.

make menuconfig

Search for 'eudev' (hit '/' - this opens the search dialog) or directly browse to Target packages -> Hardware handling and enable eudev.

kernel config

Now we need the p9 support in the kernel.

make linux-menuconfig

AFAICT these are all the configuration options required:

CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_9P_FS=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_BLK_MQ_VIRTIO=y

build

make BR2_JLEVEL=9

BR2_JLEVEL=9 is fine for my CPU, as I have 2 Cores with 2 Threads = 4 Threads (total) and once I read on the internets that it's kinda optimal to use #threads*2 + 1 make-jobs.

qemu

distro-provided qemu

I first started with the qemu-system-mips provided from my distribution, but that one gave the following error when starting the system.

request_module: runaway loop modprobe binfmt-4c46
Starting init: /sbin/init exists but couldn't execute it (error -8)
request_module: runaway loop modprobe binfmt-4c46
Starting init: /bin/sh exists but couldn't execute it (error -8)

So I went to build QEMU from source.

qemu built from source

cd -
git clone --branch v2.11.0 --depth 1 git://git.qemu-project.org/qemu.git

I chose to go with the latest release of QEMU at the time of writing, not the latest version from the repository.

The following probably requires apt install libcap-dev libattr1-dev.

mkdir qemu_mips32
cd qemu_mips32/
../qemu/configure --target-list=mips-softmmu --enable-virtfs

Also we're again doing an out-of-tree build to keep the repo clean.

make -j9

run

A minimal example on how to run a specific Buildroot-build for QEMU is contained in <buildroot>/board/qemu/<your-chosen-build>/readme.txt.

cd -
cd buildroot_mips32be/

We also want to pass a local folder to the Buildroot-build which leads us to the following commandline (replace /path/to/your/stuff/you/want/to/share/ before running this):

../qemu_mips32/mips-softmmu/qemu-system-mips -M malta -cpu mips32r6-generic -kernel images/vmlinux -drive file=images/rootfs.ext2,format=raw  -append "root=/dev/hda" -fsdev local,id=host0,path=/path/to/your/stuff/you/want/to/share/,security_model=passthrough -device virtio-9p-pci,fsdev=host0,mount_tag=host0 -nographic

terminate a running qemu instance

If you want to terminate a running QEMU guest immediately you can invoke the following command:

CTRL-A + X

running your application

cross-compile your application

Well that's a tutorial for the libtom projects and most of the projects can be built by passing the CROSS_COMPILE make- or environment-variable.

You need to have the absolute path to your buildroot_mips32be folder ready and then it's only a matter of:

make -j9 CROSS_COMPILE=/path/to/buildroot_mips32be/host/bin/mips-buildroot-linux-uclibc- <build-target>

Now you can copy this output to the /path/to/your/stuff/you/want/to/share/ you gave when starting the QEMU guest and you should be able to use it there.

mount the shared folder

As an easy example we're mounting the plan9 nfs-share to a folder (this has to be done in the guest).

mkdir /tmp/share
mount host0 /tmp/share -t 9p -o trans=virtio

This means now that the /tmp/share/ folder on the QEMU guest should contain the content of the /path/to/your/stuff/you/want/to/share/ folder on your host pc.