Installing Arch Linux ARM(aarch64) with ZFS root on RPi4

Prerequisite

Arch Linux aarch64 live system. Refer https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-4 .
USB Keyboard
HDMI cable(or access to serial console).

Boot up your aarch64 Arch Linux. Plug in SD card adapter and ensure that disk shows up on your system.

lsblk
# find your sd card block device.

Assuming sd card device is /dev/sda. Change below script contents according to your environment.

Upgrade your system

sudo pacman -Syu

Install zfs-dkms on live system

We have to create zpool on sd card, so zfs module is needed. Install yay(by normal user who can use sudo).

sudo pacman -S --needed git base-devel
git clone https://aur.archlinux.org/yay.git
cd yay
makepkg -si

# cleanup. this is optional.
cd ..
rm -rf yay

Install zfs-dkms.

yay -Syu zfs-dkms zfs-utils

# ignore architecture warnings. this will take a while...

Switch to root.

sudo su
cd

Prepare partitions

Make two partitions. First partition will be used as boot partition, formatted with vfat. Second partition will be the ZFS root.

fdisk /dev/sda

# type below commands...
o
p
n
p
1
<enter>
+512M
t
c
n
p
2
<enter> <enter>
w

Format your boot partition.

mkfs.vfat /dev/sda1

Setup dm-crypt on second partition.

cryptsetup --type=luks2 \
	--cipher=aes-xts-plain64 \
	--key-size=256 luksFormat "/dev/sda2"
  
cryptsetup open "/dev/sda2" cryptroot

Create zpool

export DISKID="$(ls /dev/disk/by-id/*uuid*cryptroot)"
mkdir -p /mnt

# create zpool. we will use compression=zstd option for zroot pool. change options and pool name as you prefer.
zpool create -f -o ashift=12         \
             -O acltype=posixacl       \
             -O relatime=on            \
             -O xattr=sa               \
             -O dnodesize=legacy       \
             -O normalization=formD    \
             -O mountpoint=none        \
             -O canmount=off           \
             -O devices=off            \
             -R /mnt                   \
             -O compression=zstd       \
             zroot "$DISKID"

Create datasets

zfs create -o mountpoint=none zroot/data
zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/default
zfs create -o mountpoint=/home zroot/data/home

Change the above commands as you prefer. All commands below will accord the above datasets.

# export, import, mount
zpool export zroot
zpool import -d /dev/disk/by-id -R /mnt zroot -N

zfs mount zroot/ROOT/default
zfs mount -a
mkdir -p /mnt/boot
mount /dev/sda1 /mnt/boot

zpool set bootfs=zroot/ROOT/default zroot

Setup cachefile (maybe should skip this step)

Setting the cachefile for pool is recommended way, but somehow this is not properly working on mounting at boot time on zfs-dkms. If you really want to set cachefile, use below commands.

zpool set cachefile=/etc/zfs/zpool.cache zroot
mkdir -p /mnt/etc/zfs
cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

Install Linux

pacstrap /mnt base base-devel linux-aarch64 linux-aarch64-headers linux-api-headers uboot-raspberrypi raspberrypi-bootloader uboot-tools vim sudo git

Chroot and setup root filesystem

Chroot into brand-new system.

arch-chroot /mnt

Set your locale.

vim /etc/locale.conf
vim /etc/locale.gen
locale-gen

Install required packages.

pacman-key --init
pacman-key --populate archlinuxarm

# add normal user
useradd -m -G wheel <username>
passwd <username>
EDITOR=vim visudo
# allow sudo commands for wheel group

# install yay.
su <username>
cd
git clone https://aur.archlinux.org/yay.git
cd yay
makepkg -si

# cleanup. optional.
# cd ..
# rm -rf yay

yay -S zfs-dkms zfs-utils
# ignore any architecture warnings.

# return to root user
exit

Generate initramfs

Edit /etc/mkinitcpio.conf. Use MODULES and HOOKS parameter as below:

MODULES=(pcie_brcmstb zfs)
HOOKS=(base udev autodetect modconf block keyboard encrypt zfs filesystems)

Order matters. encrypt should be after block, and zfs should be after encrypt hook and before filesystems hook. fsck hook is not needed because ZFS supports online scrubbing and self-healing. pcie_brcmstb is needed for enabling USB interfaces at boot time.

Edit /etc/default/zfs. Apply change as below.

ZPOOL_IMPORT_ALL_VISIBLE='yes'

This will import all visible zpools on boot time. This can be a security issue, but cachefile is not working at all on zfs-dkms, so use this parameter as temporary workaround. Now generate initramfs.

mkinitcpio -P

Modify kernel parameter

Arch Linux aarch64 uses U-Boot as a bootloader. You should modify kernel parameter according to your settings. You can tweak around on /etc/boot.txt. Example:

part uuid ${devtype} ${devnum}:2 uuid

setenv bootargs console=ttyS1,115200 console=tty0 cryptdevice=UUID=<your encrypted partition uuid>:cryptroot: zfs=zroot/ROOT/default rw smsc95xx.macaddr="${usbethaddr}"

if load ${devtype} ${devnum}:${bootpart} ${kernel_addr_r} /Image; then
  if load ${devtype} ${devnum}:${bootpart} ${fdt_addr_r} /dtbs/${fdtfile}; then
    if load ${devtype} ${devnum}:${bootpart} ${ramdisk_addr_r} /initramfs-linux.img; then
      booti ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r};
    else
      booti ${kernel_addr_r} - ${fdt_addr_r};
    fi;
  fi;
fi

Run ./mkscr.

Setup /etc/fstab

Edit /etc/fstab. You can use genfstab -U -p /mnt >> /mnt/etc/fstab outside of chroot, but this is maybe overkill for ZFS. Probably you can use genfstab and opt out automounting zfs datasets and change disk selectors as UUID, not device name. You can find UUID with lsblk -f. Example:

# zroot/ROOT/default
zroot/ROOT/default  	/         	zfs       	rw,nodev,xattr,posixacl	0 0

# /boot
UUID=<boot partition UUID>           	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro	0 2

zroot/ROOT/default should be listed because it was not automounting dataset.

Enable services

On chroot environment,

systemctl enable zfs.target
systemctl enable zfs-mount

Export pool

Exit chroot environment. And use below commands:

umount /mnt/boot
zfs umount -a
zfs umount zroot/ROOT/default
zpool export zroot

cryptsetup close cryptroot
sync

Finalize

Shutdown your live system, extract SD card, and try to boot it up.

skystar-p/rpi4_zfs.md