Raspberry Pi: How to cross-compile and use Mainline Kernel

mainline_guide.md

Raspberry Pi: How to cross-compile and use Mainline Kernel

Notes on how to set up a new Ubuntu LTS x64 environment, how to build a recent Mainline Kernel and place it on a Raspberry Pi OS SD card.

Procedure

1. Install tools needed:

$ apt install git make gcc g++ device-tree-compiler bc bison flex libssl-dev libncurses-dev python3-ply python3-git libgmp3-dev libmpc-dev

2. Install an up-to-date cross compiler and associated toolset. This may be obtained from https://snapshots.linaro.org/gnu-toolchain/. The directory arm-linux-gnueabihf contains the necessary compiler for ARM 32-bit and the directory aarch64-linux-gnu for ARM 64-bit.

Choose the version suited to your development machine's architecture. For example, at the present time, for use on 64-bit Ubuntu on an Intel-based development machine, the appropriate version is gcc-linaro-11.3.1-2022.06-x86_64_arm-linux-gnueabihf.tar.xz.

$ wget https://snapshots.linaro.org/gnu-toolchain/11.3-2022.06-1/arm-linux-gnueabihf/gcc-linaro-11.3.1-2022.06-x86_64_arm-linux-gnueabihf.tar.xz
$ sudo tar xf gcc-linaro-11.3.1-2022.06-x86_64_arm-linux-gnueabihf.tar.xz -C /opt

3. Get Mainline Linux Sources (try to avoid linux-next which tends to be unstable)

$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

4. Move into the repository folder and setup cross-compiler (for ARM 64-bit replace ARCH=arm with ARCH=arm64):

$ cd linux
$ export ARCH=arm
$ export CROSS_COMPILE=/opt/gcc-linaro-11.3.1-2022.06-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-

5. Choose the kernel configuration. To do this, you need to know what model of System on a Chip (SoC) is used on your Pi. Please refer to the Raspberry Pi Wikipedia Section. Since we are building from an upstream source, then, by reference to Emma Anholt's pages we can deduce that we should select bcm2835_defconfig for BCM2835-based models, multi_v7_defconfig for BCM2836- and BCM2837-based models (ARM 32-bit), multi_v7_lpae_defconfig for BCM2711-based models (ARM 32-bit LPAE) or defconfig for BCM2837- and BCM2711-based models (ARM64). For the Raspberry Pi 3B Plus (ARM 32-bit), the following is appropriate:

$ make multi_v7_defconfig

6. At this point you can modify the configuration using, for instance, menuconfig:

$ make menuconfig

7. Build the kernel, the modules and the device tree blobs:

$ make

To speed up compilation on a multicore machine, add the argument -j <number_of_cores+1> to the above command.

8. Insert an existing Raspberry Pi OS SD Card to your development machine. Please keep in mind that kernel architecture must match accordingly (Raspberry Pi OS 32 bit for arm and Raspberry Pi OS 64 bit for arm64). In Ubuntu, it will appear at /media/<username>/bootfs and /media/<username>/rootfs, where <username> is the username you are logged in under.
9. Copy the kernel file to the card's boot directory (for ARM 64-bit the kernel file is arch/arm64/boot/Image ):

$ cp arch/arm/boot/zImage /media/<username>/bootfs/

10. Copy the device tree blob to the card's boot directory. The exact device tree blob file to use depends on the model. The following is for a Raspberry Pi 3B Plus, so the device tree blob's filename is bcm2837-rpi-3-b-plus.dtb (for ARM 64-bit the source directory is arch/arm64/boot/dts/broadcom):

$ cp arch/arm/boot/dts/broadcom/bcm2837-rpi-3-b-plus.dtb /media/<username>/bootfs/

11. Adjust the device tree information in /media/<username>/bootfs/config.txt:

• Add the following lines to /media/<username>/bootfs/config.txt. The filename is that of the device tree blob copied over in Step 10. If /media/<username>/bootfs/config.txt already contains a device_tree entry, replace it with this. For ARM 64-bit the kernel is Image.

device_tree=bcm2837-rpi-3-b-plus.dtb
kernel=zImage

• In case you need to debug kernel issues during early boot stage, try the following settings according to your setup:

Board	console= (cmdline.txt)	earlycon= (cmdline.txt)	enable_uart= (config.txt)
Raspberry Pi 3	ttyS1,115200	uart8250,mmio32,0x3f215040	0

12. Copy the kernel modules to the SD card. Note that you need to be in superuser mode for this (for ARM 64-bit replace ARCH=arm with ARCH=arm64):

$ sudo ARCH=arm make modules_install INSTALL_MOD_PATH=/media/<username>/rootfs

13. Safely dismount the SD card from the development machine, put it into the target machine and reboot.

Notes

• Always use the DTB files from the mainline tree and not the downstream ones
• Avoid using rpi-update, as it will overwrite your mainline binary. The downstream kernel typically has problems with the upstream device tree blob.
• In some cases (e.g. the arm64), the kernel image tends to be very big (~ 16 MB) and may hit the limit of the bootfs partition.
• The AUX UART has a different device name with mainline kernel: /dev/ttyS1 instead of /dev/ttyS0

It definitely works with Raspbian 64-bit, but a single mistake result into a non-working setup and there might be new issues. Already mentioned in the notes the defconfig 64-bit image is a lot larger than the optimized image.

So based on your remarks it's hard to narrow down the issue, but something fundamental is wrong because of the rainbow screen:
What toolchain did you use?
Which kernel tree / version did you use?
Did you replaced the bcm2711-rpi-4-b.dtb with the mainline one?
Did you replaced the kernel8.img or specified the kernel within config.txt?

I followed https://gist.github.com/G-UK/ee7edc4844f14fec12450b2211fc886e and managed to make it work. Raspberry Pi 4B 64-bit should use kernel config set bcm2711_defconfig instead of defconfig.

But this is the vendor kernel. In mainline there is no bcm2711_defconfig.

Sorry, what is mainline (my poor English)? I just want to modify on top of the vendor kernel to allow certain kernel-hack features such as kexec, don't want to change too much really.

No problem

mainline = the kernel tree of Linus Torvalds
vendor tree = older version of the Torvalds tree with a lot of vendor modifications

Hi @lategoodbye does this work for rPi4 32bit on 6.0RC1?

Yes

It worked with 6.0rc1 on rpi4. I copied over a bcm2711 device blob tree. Thanks @lategoodbye

Oh, could you update the linaro url to https://snapshots.linaro.org/gnu-toolchain/
They are putting new releases there now

Thanks for pointing out. Done