Build and run minimal Linux / Busybox systems in Qemu

Linux 内核及内核模块开发环境搭建

## Common

````
export OPT=/opt
export BUILDS=/some/where/mini_linux
mkdir -p $BUILDS
````

## Linux kernel

````
export LINUX=$OPT/linux
export LINUX_BUILD=$BUILDS/linux
mkdir -p $LINUX_BUILD
cd $LINUX
make O=$LINUX_BUILD allnoconfig
cd $LINUX_BUILD
make menuconfig
````

Configure the kernel according the following:

````
64-bit kernel ---> yes
General setup ---> Initial RAM filesystem and RAM disk (initramfs/initrd) support ---> yes
General setup ---> Configure standard kernel features ---> Enable support for printk ---> yes
Executable file formats / Emulations ---> Kernel support for ELF binaries ---> yes
Executable file formats / Emulations ---> Kernel support for scripts starting with #! ---> yes
Device Drivers ---> Generic Driver Options ---> Maintain a devtmpfs filesystem to mount at /dev ---> yes
Device Drivers ---> Generic Driver Options ---> Automount devtmpfs at /dev, after the kernel mounted the rootfs ---> yes
Device Drivers ---> Character devices ---> Enable TTY ---> yes
Device Drivers ---> Character devices ---> Serial drivers ---> 8250/16550 and compatible serial support ---> yes
Device Drivers ---> Character devices ---> Serial drivers ---> Console on 8250/16550 and compatible serial port ---> yes
File systems ---> Pseudo filesystems ---> /proc file system support ---> yes
File systems ---> Pseudo filesystems ---> sysfs file system support ---> yes
````

Build the kernel:

````
time make -j8
````

````
...
Kernel: arch/x86/boot/bzImage is ready  (#1)

real    2m37.247s
user    1m58.541s
sys     0m25.542s
````

## Busybox

````
export BUSYBOX=$OPT/busybox
export BUSYBOX_BUILD=$BUILDS/busybox
mkdir -p $BUSYBOX_BUILD
cd $BUSYBOX
make O=$BUSYBOX_BUILD defconfig
cd $BUSYBOX_BUILD
make menuconfig
````

Configure Busybox according the following:

````
Busybox Settings ---> Build Options ---> Build BusyBox as a static binary (no shared libs) ---> yes
````

Build Busybox:

````
time make -j8
````

````
...
Final link with: m

real    0m20.356s
user    0m46.959s
sys     0m10.628s
````

Install Busybox:

````
make install
````

Create an initramfs:

````
export INITRAMFS_BUILD=$BUILDS/initramfs
mkdir -p $INITRAMFS_BUILD
cd $INITRAMFS_BUILD
mkdir -p bin sbin etc proc sys usr/bin usr/sbin
cp -a $BUSYBOX_BUILD/_install/* .
````

Add a `$INITRAMFS_BUILD/init` script to the initramfs with the following content:

````
#!/bin/sh

mount -t proc none /proc
mount -t sysfs none /sys

cat <<!


Boot took $(cut -d' ' -f1 /proc/uptime) seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


!
exec /bin/sh
````

Create the initramfs archive:

````
chmod +x init
find . -print0 | cpio --null -ov --format=newc \
  | gzip -9 > $BUILDS/initramfs.cpio.gz
````

## Run and see (`<CTRL>a x` to quit)

````
qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDS/initramfs.cpio.gz -nographic \
  -append "console=ttyS0"
````

Note: for better performance, add the `-enable-kvm` option if your host has KVM enabled:
````
qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDS/initramfs.cpio.gz -nographic \
  -append "console=ttyS0" -enable-kvm
````

````
...


Boot took 0.45 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls /
bin      etc      linuxrc  root     sys
dev      init     proc     sbin     usr
/ # QEMU: Terminated
````

## Buildroot

We assume that a toolchain is available in `/opt/toolchains/x86_64-unknown-linux-gnu` with prefix `x86_64-unknown-linux-gnu`, gcc version 5.x, kernel headers series 4.3.x, glibc C library and C++ support. These are reasonable defaults if you are using a toolchain generated by crosstool-NG. Adapt to your own situation. Notes:
* You cannot use the native toolchain of your host computer (see Buildroot documentation to understand why).
* If you do not have a toolchain already, you can build one using crosstool-NG (or Buildroot itself) and reuse it for other projects.
* crosstool-NG is the recommended tool to build your own toolchain but avoid using uClibc (no IPV6 support), prefer uClibc-ng or glibc.
* You can also use the built-in toolchain of Buildroot but be aware that it will take way longer than using an existing toolchain. Be also aware that in many cases you will have to re-build the toolchain after modifying the Buildroot configuration.
* No yet convinced? Please use crosstool-NG, build and use your own toolchain.

````
export BUILDROOT=$OPT/buildroot
export BUILDROOT_BUILD=$BUILDS/buildroot
mkdir -p $BUILDROOT_BUILD
cd $BUILDROOT_BUILD
touch Config.in external.mk
echo 'name: mini_linux' > external.desc
echo 'desc: minimal linux system with buildroot' >> external.desc
mkdir configs overlay
cd $BUILDROOT
make O=$BUILDROOT_BUILD BR2_EXTERNAL=$BUILDROOT_BUILD qemu_x86_64_defconfig
cd $BUILDROOT_BUILD
make menuconfig
````

Configure Buildroot according the following:

````
Build options ---> Location to save buildroot config ---> $(BR2_EXTERNAL)/configs/mini_linux_defconfig
Build options ---> Download dir ---> /some/where/buildroot_dl
Build options ---> Number of jobs to run simultaneously (0 for auto) ---> 8
Build options ---> Enable compiler cache ---> yes
Build options ---> Compiler cache location ---> /some/where/buildroot_ccache
Toolchain ---> Toolchain type ---> External toolchain
Toolchain ---> Toolchain ---> Custom toolchain
Toolchain ---> Toolchain origin ---> Pre-installed toolchain
Toolchain ---> Toolchain path ---> /opt/toolchains/x86_64-unknown-linux-gnu
Toolchain ---> Toolchain prefix ---> x86_64-unknown-linux-gnu
Toolchain ---> External toolchain gcc version ---> 5.x
Toolchain ---> External toolchain kernel headers series ---> 4.3.x
Toolchain ---> External toolchain C library ---> glibc/eglibc
Toolchain ---> Toolchain has C++ support? ---> yes
System configuration ---> System hostname ---> mini_linux
System configuration ---> System banner ---> Welcome to mini_linux
System configuration ---> Run a getty (login prompt) after boot ---> TTY port ---> ttyS0
System configuration ---> Network interface to configure through DHCP --->
System configuration ---> Root filesystem overlay directories ---> $(BR2_EXTERNAL)/overlay
Kernel ---> Linux Kernel ---> no
Filesystem images ---> cpio the root filesystem (for use as an initial RAM filesystem) ---> yes
Filesystem images ---> Compression method ---> gzip
````

Save the configuration and build:

````
make savedefconfig
````

Add a `$BUILDROOT_BUILD/overlay/init` script to the overlay with the following content:

````
#!/bin/sh
/bin/mount -t devtmpfs devtmpfs /dev
/bin/mount -t proc none /proc
/bin/mount -t sysfs none /sys
exec 0</dev/console
exec 1>/dev/console
exec 2>/dev/console
cat <<!


Boot took $(cut -d' ' -f1 /proc/uptime) seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


!
exec /bin/sh
````

Build the root filesystem:

````
chmod +x overlay/init
time make
````

````
...
real    1m52.905s
user    0m50.682s
sys     0m36.928s
````

## Run and see (`<CTRL>a x` to quit)

````
qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDROOT_BUILD/images/rootfs.cpio.gz -nographic \
  -append "console=ttyS0"
````

Note: for better performance, add the `-enable-kvm` option if your host has KVM enabled.

````
...


Boot took 0.57 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls /
bin      init     linuxrc  opt      run      tmp
dev      lib      media    proc     sbin     usr
etc      lib64    mnt      root     sys      var
/ # QEMU: Terminated
````

## Add and run a custom user application

Create a new directory for the custom user applications:

````
export APPS=$BUILDS/apps
mkdir -p $APPS
cd $APPS
````

Add an application source file `$APPS/hello_world.c` with the following content:

````
#include <stdio.h>

int main(int argc, char **argv) {
	printf("mini_linux says: Hello world!\n");
	return 0;
}
````

Add a `$APPS/Makefile` with the following content (replace the `CROSS_COMPILE` definition with whatever is appropriate in your specific case):

````
CROSS_COMPILE	:= /opt/toolchains/x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-
CC		:= $(CROSS_COMPILE)gcc

hello_world: hello_world.o
	$(CC) -o $@ $<

hello_world.o: hello_world.c
	$(CC) -c -o $@ $<

clean:
	rm -f hello_world hello_world.o
````

Compile the application, copy it in the Buildroot overlay directory and re-build the root filesystem:

````
make
...
cp hello_world $BUILDROOT_BUILD/overlay
...
cd $BUILDROOT_BUILD
make
````

## Run and see (`<CTRL>a x` to quit)

````
qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDROOT_BUILD/images/rootfs.cpio.gz -nographic \
  -append "console=ttyS0"
````

Note: for better performance, add the `-enable-kvm` option if your host has KVM enabled.

````
...


Boot took 0.57 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls
bin          init         media        root         tmp
dev          lib          mnt          run          usr
etc          lib64        opt          sbin         var
hello_world  linuxrc      proc         sys
/ # ./hello_world
mini_linux says: Hello world!
/ # QEMU: Terminated
````

## Add loadable module support to the Linux kernel

````
cd $LINUX_BUILD
make menuconfig
````

Change the kernel configuration according the following:

````
Enable loadable module support ---> yes
````

Re-build the kernel and its modules (none, in our case) and install the modules in the Buildroot overlay directory:

````
make -j8
make -j8 modules
make modules_install INSTALL_MOD_PATH=$BUILDROOT_BUILD/overlay
````

## Add a custom user module

Create a new directory for the custom user modules:

````
export MODULES=$BUILDS/modules
mkdir -p $MODULES
cd $MODULES
````

Add a module source file `$MODULES/hello_world.c` with the following content:

````
/* hello_world.c */
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

static int __init first_init(void)
{
  pr_info("mini_linux module says: Hello world!\n");
  return 0;
}

static void __exit first_exit(void)
{
  pr_info("Bye\n");
}

module_init(first_init);
module_exit(first_exit);

MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("My first module");
MODULE_AUTHOR("The Doctor");
````

Add a `$MODULES/Makefile` with the following content:

````
ifneq ($(KERNELRELEASE),)
# kbuild part of makefile
obj-m := hello_world.o

else
# normal makefile
KDIR ?= $(LINUX_BUILD)

default:
	$(MAKE) -C $(KDIR) M=$$PWD

modules_install:
	$(MAKE) -C $(KDIR) M=$$PWD $@

clean:
	rm -rf *.o .*.cmd *.ko hello_world.mod.c modules.order Module.symvers .tmp_versions
endif
````

Compile the module, install it in the Buildroot overlay directory and re-build the root filesystem:

````
make
...
make modules_install INSTALL_MOD_PATH=$BUILDROOT_BUILD/overlay
...
cd $BUILDROOT_BUILD
make
````

## Run and see (`<CTRL>a x` to quit)

````
qemu-system-x86_64 -kernel $LINUX_BUILD/arch/x86_64/boot/bzImage \
  -initrd $BUILDROOT_BUILD/images/rootfs.cpio.gz -nographic \
  -append "console=ttyS0"
````

Note: for better performance, add the `-enable-kvm` option if your host has KVM enabled.

````
...


Boot took 0.57 seconds

        _       _     __ _                  
  /\/\ (_)_ __ (_)   / /(_)_ __  _   ___  __
 /    \| | '_ \| |  / / | | '_ \| | | \ \/ /
/ /\/\ \ | | | | | / /__| | | | | |_| |>  < 
\/    \/_|_| |_|_| \____/_|_| |_|\__,_/_/\_\ 


Welcome to mini_linux


/ # ls lib/modules/4.8.0\+/extra
hello_world.ko
/ # lsmod
Module                  Size  Used by    Not tainted
/ # insmod lib/modules/4.8.0\+/extra/hello_world.ko
hello_world: loading out-of-tree module taints kernel.
mini_linux module says: Hello world!
/ # lsmod
Module                  Size  Used by    Tainted: G  
hello_world              704  -
/ # QEMU: Terminated
````

Linux 系统由四部分组成：

1. Linux 内核
2. Shell (busybox)
3. 文件系统 (initramfs)
4. 应用程序 (optional)