Skip to content

Instantly share code, notes, and snippets.

@chrisdone

chrisdone/gist:02e165a0004be33734ac2334f215380e

Last active October 4, 2024 00:07
Show Gist options
  • Save chrisdone/02e165a0004be33734ac2334f215380e to your computer and use it in GitHub Desktop.
Save chrisdone/02e165a0004be33734ac2334f215380e to your computer and use it in GitHub Desktop.
Download ZIP
Build and run minimal Linux / Busybox systems in Qemu
Raw
gistfile1.md

Common

export OPT=/opt
export BUILDS=/some/where/mini_linux
mkdir -p $BUILDS

Linux kernel

export LINUX=$OPT/linux
export LINUX_BUILD=$BUILDS/linux
mkdir -p $LINUX_BUILD
cd $LINUX
make O=$LINUX_BUILD allnoconfig
cd $LINUX_BUILD
make menuconfig

Configure the kernel according the following:

64-bit kernel ---> yes
General setup ---> Initial RAM filesystem and RAM disk (initramfs/initrd) support ---> yes
General setup ---> Configure standard kernel features ---> Enable support for printk ---> yes
Executable file formats / Emulations ---> Kernel support for ELF binaries ---> yes
Executable file formats / Emulations ---> Kernel support for scripts starting with #! ---> yes
Device Drivers ---> Generic Driver Options ---> Maintain a devtmpfs filesystem to mount at /dev ---> yes
Device Drivers ---> Generic Driver Options ---> Automount devtmpfs at /dev, after the kernel mounted the rootfs ---> yes
Device Drivers ---> Character devices ---> Enable TTY ---> yes
Device Drivers ---> Character devices ---> Serial drivers ---> 8250/16550 and compatible serial support ---> yes
Device Drivers ---> Character devices ---> Serial drivers ---> Console on 8250/16550 and compatible serial port ---> yes
File systems ---> Pseudo filesystems ---> /proc file system support ---> yes
File systems ---> Pseudo filesystems ---> sysfs file system support ---> yes

Build the kernel:

time make -j8

...
Kernel: arch/x86/boot/bzImage is ready  (#1)

real    2m37.247s
user    1m58.541s
sys     0m25.542s

Busybox

export BUSYBOX=$OPT/busybox
export BUSYBOX_BUILD=$BUILDS/busybox
mkdir -p $BUSYBOX_BUILD
cd $BUSYBOX
make O=$BUSYBOX_BUILD defconfig
cd $BUSYBOX_BUILD
make menuconfig

Configure Busybox according the following:

Busybox Settings ---> Build Options ---> Build BusyBox as a static binary (no shared libs) ---> yes

Build Busybox:

time make -j8

...
Final link with: m

real    0m20.356s
user    0m46.959s
sys     0m10.628s

Install Busybox:

make install

Create an initramfs:

export INITRAMFS_BUILD=$BUILDS/initramfs
mkdir -p $INITRAMFS_BUILD
cd $INITRAMFS_BUILD
mkdir -p bin sbin etc proc sys usr/bin usr/sbin
cp -a $BUSYBOX_BUILD/_install/* .

Add a $INITRAMFS_BUILD/init script to the initramfs with the following content:

#!/bin/sh

mount -t proc none /proc
mount -t sysfs none /sys

cat <<!


Boot took $(cut -d' ' -f1 /proc/uptime) seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


!
exec /bin/sh

Create the initramfs archive:

chmod +x init
find . -print0 | cpio --null -ov --format=newc \
  | gzip -9 > $BUILDS/initramfs.cpio.gz

Run and see (<CTRL>a x to quit)

qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDS/initramfs.cpio.gz -nographic \
  -append "console=ttyS0"

Note: for better performance, add the -enable-kvm option if your host has KVM enabled:

qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDS/initramfs.cpio.gz -nographic \
  -append "console=ttyS0" -enable-kvm

...


Boot took 0.45 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls /
bin      etc      linuxrc  root     sys
dev      init     proc     sbin     usr
/ # QEMU: Terminated

Buildroot

We assume that a toolchain is available in /opt/toolchains/x86_64-unknown-linux-gnu with prefix x86_64-unknown-linux-gnu, gcc version 5.x, kernel headers series 4.3.x, glibc C library and C++ support. These are reasonable defaults if you are using a toolchain generated by crosstool-NG. Adapt to your own situation. Notes:

  • You cannot use the native toolchain of your host computer (see Buildroot documentation to understand why).
  • If you do not have a toolchain already, you can build one using crosstool-NG (or Buildroot itself) and reuse it for other projects.
  • crosstool-NG is the recommended tool to build your own toolchain but avoid using uClibc (no IPV6 support), prefer uClibc-ng or glibc.
  • You can also use the built-in toolchain of Buildroot but be aware that it will take way longer than using an existing toolchain. Be also aware that in many cases you will have to re-build the toolchain after modifying the Buildroot configuration.
  • No yet convinced? Please use crosstool-NG, build and use your own toolchain.
export BUILDROOT=$OPT/buildroot
export BUILDROOT_BUILD=$BUILDS/buildroot
mkdir -p $BUILDROOT_BUILD
cd $BUILDROOT_BUILD
touch Config.in external.mk
echo 'name: mini_linux' > external.desc
echo 'desc: minimal linux system with buildroot' >> external.desc
mkdir configs overlay
cd $BUILDROOT
make O=$BUILDROOT_BUILD BR2_EXTERNAL=$BUILDROOT_BUILD qemu_x86_64_defconfig
cd $BUILDROOT_BUILD
make menuconfig

Configure Buildroot according the following:

Build options ---> Location to save buildroot config ---> $(BR2_EXTERNAL)/configs/mini_linux_defconfig
Build options ---> Download dir ---> /some/where/buildroot_dl
Build options ---> Number of jobs to run simultaneously (0 for auto) ---> 8
Build options ---> Enable compiler cache ---> yes
Build options ---> Compiler cache location ---> /some/where/buildroot_ccache
Toolchain ---> Toolchain type ---> External toolchain
Toolchain ---> Toolchain ---> Custom toolchain
Toolchain ---> Toolchain origin ---> Pre-installed toolchain
Toolchain ---> Toolchain path ---> /opt/toolchains/x86_64-unknown-linux-gnu
Toolchain ---> Toolchain prefix ---> x86_64-unknown-linux-gnu
Toolchain ---> External toolchain gcc version ---> 5.x
Toolchain ---> External toolchain kernel headers series ---> 4.3.x
Toolchain ---> External toolchain C library ---> glibc/eglibc
Toolchain ---> Toolchain has C++ support? ---> yes
System configuration ---> System hostname ---> mini_linux
System configuration ---> System banner ---> Welcome to mini_linux
System configuration ---> Run a getty (login prompt) after boot ---> TTY port ---> ttyS0
System configuration ---> Network interface to configure through DHCP --->
System configuration ---> Root filesystem overlay directories ---> $(BR2_EXTERNAL)/overlay
Kernel ---> Linux Kernel ---> no
Filesystem images ---> cpio the root filesystem (for use as an initial RAM filesystem) ---> yes
Filesystem images ---> Compression method ---> gzip

Save the configuration and build:

make savedefconfig

Add a $BUILDROOT_BUILD/overlay/init script to the overlay with the following content:

#!/bin/sh
/bin/mount -t devtmpfs devtmpfs /dev
/bin/mount -t proc none /proc
/bin/mount -t sysfs none /sys
exec 0</dev/console
exec 1>/dev/console
exec 2>/dev/console
cat <<!


Boot took $(cut -d' ' -f1 /proc/uptime) seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


!
exec /bin/sh

Build the root filesystem:

chmod +x overlay/init
time make

...
real    1m52.905s
user    0m50.682s
sys     0m36.928s

Run and see (<CTRL>a x to quit)

qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDROOT_BUILD/images/rootfs.cpio.gz -nographic \
  -append "console=ttyS0"

Note: for better performance, add the -enable-kvm option if your host has KVM enabled.

...


Boot took 0.57 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls /
bin      init     linuxrc  opt      run      tmp
dev      lib      media    proc     sbin     usr
etc      lib64    mnt      root     sys      var
/ # QEMU: Terminated

Add and run a custom user application

Create a new directory for the custom user applications:

export APPS=$BUILDS/apps
mkdir -p $APPS
cd $APPS

Add an application source file $APPS/hello_world.c with the following content:

#include <stdio.h>

int main(int argc, char **argv) {
	printf("mini_linux says: Hello world!\n");
	return 0;
}

Add a $APPS/Makefile with the following content (replace the CROSS_COMPILE definition with whatever is appropriate in your specific case):

CROSS_COMPILE	:= /opt/toolchains/x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-
CC		:= $(CROSS_COMPILE)gcc

hello_world: hello_world.o
	$(CC) -o $@ $<

hello_world.o: hello_world.c
	$(CC) -c -o $@ $<

clean:
	rm -f hello_world hello_world.o

Compile the application, copy it in the Buildroot overlay directory and re-build the root filesystem:

make
...
cp hello_world $BUILDROOT_BUILD/overlay
...
cd $BUILDROOT_BUILD
make

Run and see (<CTRL>a x to quit)

qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDROOT_BUILD/images/rootfs.cpio.gz -nographic \
  -append "console=ttyS0"

Note: for better performance, add the -enable-kvm option if your host has KVM enabled.

...


Boot took 0.57 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls
bin          init         media        root         tmp
dev          lib          mnt          run          usr
etc          lib64        opt          sbin         var
hello_world  linuxrc      proc         sys
/ # ./hello_world
mini_linux says: Hello world!
/ # QEMU: Terminated

Add loadable module support to the Linux kernel

cd $LINUX_BUILD
make menuconfig

Change the kernel configuration according the following:

Enable loadable module support ---> yes

Re-build the kernel and its modules (none, in our case) and install the modules in the Buildroot overlay directory:

make -j8
make -j8 modules
make modules_install INSTALL_MOD_PATH=$BUILDROOT_BUILD/overlay

Add a custom user module

Create a new directory for the custom user modules:

export MODULES=$BUILDS/modules
mkdir -p $MODULES
cd $MODULES

Add a module source file $MODULES/hello_world.c with the following content:

/* hello_world.c */
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

static int __init first_init(void)
{
  pr_info("mini_linux module says: Hello world!\n");
  return 0;
}

static void __exit first_exit(void)
{
  pr_info("Bye\n");
}

module_init(first_init);
module_exit(first_exit);

MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("My first module");
MODULE_AUTHOR("The Doctor");

Add a $MODULES/Makefile with the following content:

ifneq ($(KERNELRELEASE),)
# kbuild part of makefile
obj-m := hello_world.o

else
# normal makefile
KDIR ?= $(LINUX_BUILD)

default:
	$(MAKE) -C $(KDIR) M=$$PWD

modules_install:
	$(MAKE) -C $(KDIR) M=$$PWD $@

clean:
	rm -rf *.o .*.cmd *.ko hello_world.mod.c modules.order Module.symvers .tmp_versions
endif

Compile the module, install it in the Buildroot overlay directory and re-build the root filesystem:

make
...
make modules_install INSTALL_MOD_PATH=$BUILDROOT_BUILD/overlay
...
cd $BUILDROOT_BUILD
make

Run and see (<CTRL>a x to quit)

qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDROOT_BUILD/images/rootfs.cpio.gz -nographic \
  -append "console=ttyS0"

Note: for better performance, add the -enable-kvm option if your host has KVM enabled.

...


Boot took 0.57 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls lib/modules/4.8.0\+/extra
hello_world.ko
/ # lsmod
Module                  Size  Used by    Not tainted
/ # insmod lib/modules/4.8.0\+/extra/hello_world.ko
hello_world: loading out-of-tree module taints kernel.
mini_linux module says: Hello world!
/ # lsmod
Module                  Size  Used by    Tainted: G  
hello_world              704  -
/ # QEMU: Terminated
@cirosantilli
Copy link

Here is a highly automated version of this that might be of interest: https://github.com/cirosantilli/linux-kernel-module-cheat

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment