Access to the serial console is likely mandatory for this, though it could potentially be done purely via SSH. There will be no way to debug issues in case the system does not come up.
These steps describe how to ultimately boot Gentoo on the board, with the bootloader(s) in NAND and Gentoo and the Kernels on an NVMe SSD.
The most complete u-boot and kernel tree for the device can be found on frank-w's Github: https://github.com/frank-w/u-boot/ https://github.com/frank-w/BPI-Router-Linux/tree/6.9-main
I've used the 6.9-main branch for the kernel, and the default 2024.4 branch for u-boot.
Both repos come with build.sh style scripts, but specially u-boot needs a lot of manual work, since the BPi4 uses a multi-staged boot process.
Moreso, both repos need cross-compiler for both arm-linux-gnueabihf and aarch64-linux-gnu, even though the target in the end is only aarch64. crossdev
Additionally, u-boot-tools are required.
You want a self-contained itb file for u-boot, which contains the right devicetree right in itself.
The chanes I did to the kernel are minimal: https://github.com/frank-w/BPI-Router-Linux/compare/6.9-main...BtbN:BPI-Router-Linux:6.9-main
Generally, set the desired board and other settings in build.conf, and then run ./build.sh importconfig to get the proper default config for the board. Then you can run ./build.sh config for the kernels menuconfig interface for any desired changes.
To build the kernel, simply first run ./build.sh build followed by ./build.sh pack, and you will end up with all the output files in ../SD.
It by default wants to call sudo to mount a tmpfs, to stop if from doing that, remove the ramdisksize from the build.conf.
My changes:
https://github.com/frank-w/u-boot/compare/2024-04-bpi...BtbN:u-boot:2024-04-bpi
https://github.com/frank-w/u-boot/compare/mtk-atf...BtbN:u-boot:mtk-atf
The concrete steps I put into a shell script: https://github.com/BtbN/u-boot/blob/2024-04-bpi/full_build.sh
To start of, I'd build an image for an sd card (passing sd) to the full build script, and flash it on your MicroSD card.
Since it's easier to get the bootloader onto the NAND once Gentoo can initially boot off of the SD card.
To build the actual images for the NAND install, run the full build script for spi-nand.
Make sure to add the UBI=1 to the makeflags, so it can read the fib file from ubi.
Easiest way is to use OpenWRT as a live env for the installation: https://firmware-selector.openwrt.org/?version=SNAPSHOT&target=mediatek%2Ffilogic&id=bananapi_bpi-r4
Simply flash sdcard.img.gz to a MicroSD card, flip both dip-switches down, and boot it up.
I just used a serial console here to get a root shell, but if you can ssh in to get one, that'd probably also work here.
Use opkg to install some neccesary packages, like kmod-nvme, tar (busybox tar can't extract a stage3), gdisk and e2fsprogs so you can partition the disk and make an ext4 fs on the NVMe disk.
After installing the nvme kmod, the disk should show right up. Partition it with gdisk to your liking. I put a smaller boot fat32 partition at the start for the kernel(s), but in theory u-boot can also read ext4 so that's not strictly neccesary.
Extract the stage3 tarball onto your new rootfs as usual (openwrt tar does not support xattrs, but that shouldn't cause any critical issues.
Don't forget to chroot in now and do some basic setup. At the very least change the root password and generate a machine-id. If you're doing this without a serial console, also set up networking (which is quite complex on this board) and enable sshd.
Put the kernels itb file you built earlier onto your chosen boot partition and remember its path.
If you really want to do this without a serial console, put respective u-boot config for it to find and boot the kernel into the uEnv_r4.txt and build a fresh u-boot.
To actually get Gentoo booted, I used the sdcard image u-boot built, and used it to get into a u-boot shell. From there I used commands like ext4ls/ext4load to load and boot the kernels itb file.
A possible invocation would look something like this:
pci enum
nvme scan
fatload nvme 0:1 $loadaddr /6.9.0-bpi-r4-main.itb
setenv bootargs "board=bpi-r4 console=ttyS0,115200n1 earlycon=uart8250,mmio32,0x11000000 root=/dev/nvme0n1p3 rootfstype=ext4 rootwait"
bootm $loadaddr
Ideally, you are now at a Gentoo login prompt on the serial console, or can log in via ssh after figuring out the IP address somehow.
To boot from NAND, the two image spim-nand image files built by u-boot are needed.
The bl2 one goes straight to the base of the NAND. Easiest is to flash it straight from u-boot, since Linux aggressively write-protects it (/dev/mtd0):
ext4load nvme 0:3 $loadaddr /root/bpi-r4_spim-nand_bl2.img
ext4size nvme 0:3 /root/bpi-r4_spim-nand_bl2.img
mtd write spi-nand0 $loadaddr 0x0 $filesize
Then finally the fip file, which contains the actual u-boot as bl33, can be flashed from Linux using ubi tools (from sys-fs/mtd-utils):
ubiformat /dev/mtd1
ubiattach -p /dev/mtd1
ubimkvol /dev/ubi0 -N fip -s 4MiB -t static
ubiupdatevol /dev/ubi0_0 /root/bpi-r4_spim-nand_fip.bin
Some details on how those come to be can be found here: frank-w/u-boot#17 (comment)
After this, it should be possible to flick the dip-switches to NAND boot and successfully boot into Gentoo.
The boot method I came up with for myself consist of the following bootcmd baked into u-boots uEnv.txt:
bootdelay=3
bootcmd=pci enum; nvme scan; nvme info; if fatload nvme 0:1 $loadaddr boot.scr; then source $loadaddr; fi
This will simply execute a u-boot script which sits on my /boot partition.
The script looks like this:
setenv bootargs "board=bpi-r4 console=ttyS0,115200n1 earlycon=uart8250,mmio32,0x11000000 root=/dev/nvme0n1p3 rootfstype=ext4 rootwait ro"
fatload nvme 0:1 $loadaddr 6.9.0-bpi-r4-main.itb
bootm $loadaddr#conf-sd
It needs converted into an u-boot image:
mkimage -T script -O linux -A arm64 -n "Boot script" -d boot.txt boot.scr