Arch on zfs root + native encryption at rest + ZfsBootMenu (UEFI without another bootloader)

arch_on_zfs.txt

NOTE: requires an EFI bios, so zfs will not use the whole disk as this is a single disk scenario.



/*\
|* Prerequisite: preare archiso with zfs support
\*****************************************************************************/

// 0. Do everything as root.

// 1. Install archiso
pacman -S archiso

// Copy releng profile:
mkdir archlive
cp -r /usr/share/archiso/configs/releng/* archlive

// Add in archzfs repository (at the top of the list):
vim archlive/pacman.conf
  [archzfs]
  SigLevel = Never
  Server = https://github.com/archzfs/archzfs/releases/download/experimental

// Add in zfs package (x86_64 only):
vim archlive/packages.x86_64
  archzfs-linux
  linux-headers

// Build the iso:
mkarchiso -v -w /tmp/archiso-tmp -o . archlive

// Cleanup:
rm -fr archlive /tmp/archiso-tmp

// Write to usb device:
dd if=archlinux-20XX.XX.XX-x86_64.iso of=/dev/sdX status=progress



/*\
|* Install Arch
\*****************************************************************************/
// Keymap:
loadkeys <keymap>

// Check if the system is actually running in EFI mode:
ls /sys/firmware/efi/efivars

// Connect to the internet

// Update system clock:
timedatectl set-ntp true

// Create a gpt partition on disk with an EFI partition and another one for the zpool:
// NOTE: using DISK variable for convenience, set the right dev in there.
DISK="/dev/sdX"
parted -s ${DISK} mklabel gpt
parted -sa optimal ${DISK} mkpart ESP fat32 1MiB 512MiB
parted -s ${DISK} set 1 esp on
parted -sa optimal ${DISK} mkpart primary ext4 512MiB 100%

// Get the last partition's partuuid:
blkid

// Create the zfs pool:
zpool create zroot /dev/disk/by-partuuid/PARTUUID-GOTTEN-WITH-BLKID

// Set zfs cache:
zpool set cachefile=/etc/zfs/zpool.cache zroot

// Zfs tuning:
zfs set relatime=on zroot
zfs set compression=zstd zroot
zfs set checksum=blake3 zroot
zfs set mountpoint=none zroot

// Create an encrypted partition:
zfs create -o encryption=on -o keyformat=passphrase -o mountpoint=none zroot/e

// Create root zvols:
zfs create \
  -o mountpoint=none \
  -o acltype=posixacl \
  -o org.zfsbootmenu:commandline="rw zfs.zfs_arc_max=4294967296" \
  zroot/e/ROOT
zfs create -o mountpoint=/ -o canmount=noauto zroot/e/ROOT/arch

// Create home zvol:
zfs create -o mountpoint=/home zroot/e/home

// Create swap zvol:
zfs create -V 2G -b $(getconf PAGESIZE) \
  -o logbias=throughput \
  -o sync=always \
  -o primarycache=metadata \
  -o secondarycache=none \
  -o com.sun:auto-snapshot=false \
  -o compression=zle \
  zroot/e/swap
mkswap -f /dev/zvol/zroot/e/swap
swapon /dev/zvol/zroot/e/swap

// (OPTIONAL) Create tmp zvol, if you want /tmp to be persistent:
zfs create \
  -o setuid=off \
  -o devices=off \
  -o sync=disabled \
  -o mountpoint=/tmp \
  zroot/tmp
systemctl mask tmp.mount

// (OPTIONAL) Create any other needed dataset.

// Umount all zfs volumes:
swapoff -a
zpool export zroot

// Import the pool where the arch chroot will take place:
mkdir x
zpool import -R /root/x zroot

// Mount encrypted datasets:
zfs load-key zroot/e
zfs mount -la

// Setup the EFI partition as well:
mkfs.fat -F32 /dev/sdX1
mkdir -p /root/x/boot/efi
mount /dev/sdX1 /root/x/boot/efi

// Install the base system:
pacstrap x base linux linux-firmware

// Copy the zpool cache file over:
mkdir x/etc/zfs
cp /etc/zfs/zpool.cache /root/x/etc/zfs/zpool.cache

// Add in fstab data:
genfstab -U /root/x >> /root/x/etc/fstab

// Chroot:
arch-chroot x

// Vim:
pacman -S vim

// Pacman key init:
pacman-key --init
pacman-key --populate archlinux

// Add in archzfs key:
pacman-key --recv-keys 3A9917BF0DED5C13F69AC68FABEC0A1208037BE9
pacman-key --lsign-key 3A9917BF0DED5C13F69AC68FABEC0A1208037BE9

// Add in archzfs repository (at the top of the list):
vim /etc/pacman.conf
  [archzfs]
  Server = https://github.com/archzfs/archzfs/releases/download/experimental

// Install zfs stuff:
pacman -Syu
pacman -S archzfs-linux

// Create a hostid file:
zgenhostid -f $(hostid)

// Enable zfs stuff:
zpool set cachefile=/etc/zfs/zpool.cache zroot
systemctl enable zfs.target
systemctl enable zfs-import-cache
systemctl enable zfs-mount
systemctl enable zfs-import.target

// Installation settings:
// NOTE: pick the timezone you want.
timedatectl set-timezone Europe/Rome
hwclock --systohc
vim /etc/locale.gen
  enable stuff
locale-gen
vim /etc/vconsole.conf
  KEYMAP=<keymap>
  FONT=lat2-16
vim /etc/hostname
  <hostname>
vim /etc/hosts
  127.0.0.1 localhost
  ::1 localhost
  127.0.1.1 <hostname>.localdomain <hostname>
passwd

// Fix fstab:
vim /etc/fstab
  # Swap:
  /dev/zvol/zroot/e/swap none swap discard 0 0
  
  # EFI:
  UUID=<EFI PARTITION UUID> /boot/efi vfat rw,relatime,fmask=0177,dmask=0077,iocharset=utf8,shortname=mixed,errors=remount-ro,noauto 0 2
   
// Prepare the EFI partition for ZfsBootMenu:
pacman -S efibootmgr
mkdir -p /boot/efi/EFI/zbm

// Download ZfsBootMenu (check here for the latest release: https://github.com/zbm-dev/zfsbootmenu/releases )
curl -L https://github.com/zbm-dev/zfsbootmenu/releases/download/v2.2.0/zfsbootmenu-release-x86_64-v2.2.0-vmlinuz.EFI -o /boot/efi/EFI/zbm/zfsbootmenu.EFI

// Add in the new efi menu entry:
// IMPORTANT:
//  - check the value of --disk
//  - check the value of --part
//  - check the value of rd.vconsole.keymap
//  - check the value of rd.vconsole.font (lat2-16 is fine for displays smaller than 4k)
// If this fails, your pc won't boot up.
efibootmgr --disk /dev/sdX --part 1 --create --label "ZFSBootMenu" --loader '\EFI\zbm\zfsbootmenu.EFI' --unicode "spl.spl_hostid=0x$(hostid) zbm.timeout=10 zbm.prefer=zroot zbm.import_policy=hostid rd.vconsole.keymap=<keymap> rd.vconsole.font=<font> quiet" --verbose

// Safely unmount the efi partition:
umount /boot/efi

// Disclaimer: I find adding in hooks calling bash code quite tedious to maintain, and in some cases it isn't easy to specify targets.
// For this reasons I've put up a repo oh bash scripts I use: https://github.com/Soulsuke/arch-zfs-tools
// If you do not care about setting up these scripts skip to ARCH-ZFS-TOOLS-END, but you may want to read up which hooks I'm setting up as you may want/need something like them.

// Clone the repository:
cd /opt
git clone https://github.com/Soulsuke/arch-zfs-tools

// Create the pacman hooks folder:
mkdir /etc/pacman.d/hooks

// Create a pacman hook to automatically take snapshots when the linux package gets updated:
// NOTE: change target if you're using a different kernel.
vim /etc/pacman.d/hooks/00-zfs-snapshotter_root.hook
  [Trigger]
  Type = Package
  Operation = Install
  Operation = Upgrade
  Operation = Remove
  Target = linux
  
  [Trigger]
  Type = Path
  Operation = Install
  Operation = Upgrade
  Operation = Remove
  Target = usr/lib/modules/*/vmlinuz
  Target = usr/lib/initcpio/*
  
  [Action]
  Description = Creating a backup BE...
  When = PreTransaction
  Exec = /usr/bin/sh -c '/opt/arch-zfs-tools/zfs-snapshotter.bash zroot/e/ROOT/arch 3 "$(uname -r)_$(zfs --version | head -n1)"'

// ARCH-ZFS-TOOLS-END

// To avoid having to type the zfs password twice, create a passphrase file:
vim /etc/zfs/zroot.key
  <your password in plain text, no extra characters or newline at the end!>
chmod 600 /etc/zfs/zroot.key

// Set the key file for the encrypted dataset:
zfs change-key -o keylocation=file:///etc/zfs/zroot.key -o keyformat=passphrase zroot/e
// NOTE: to manually unlock the pool without the keyfile (shall you need it) you'll have to use:
//   zfs load-key -L passphrase zroot/e
// Otherwise load-key will fail as it will look for the keyfile.

// Set zfs hooks + hostid + zfs key in mkinitcpio.conf:
vim /etc/mkinitcpio.conf
  FILES=(/etc/hostid /etc/zfs/zroot.key)
  ...
  HOOKS=( ... autodetect microcode modconf kms keyboard keymap block zfs filesystems ... )
// IMPORTANT: usually bundling the key file in the initramfs is a BAD IDEA as the key becomes readable for anyone who can access the file.
//            We can safely do it here ONLY BECAUSE the resulting initramfs file will be stored on an encrypted drive.

// Generate the initramfs:
mkinitcpio -P

// Install intel-ucode if needed or (amd-ucode):
pacman -S intel-ucode

// Harden the system a little since the initramfs contains the key to unlock the pool:
chmod 700 /boot
chmod 600 /boot/*
chmod 700 /boot/efi

// Create a systemd service to perform zpool scrubs:
vim /etc/systemd/system/[email protected]
  [Unit]
  Description=Zpool scrub
  After=zfs-load-key.service
  
  [Service]
  ExecStart=/usr/bin/zpool scrub %i

// Create a systemd timer to launch scrubs periodically:
vim /etc/systemd/system/[email protected]
  [Unit]
  Description=Weekly zpool scrub
  
  [Timer]
  OnCalendar=Mon *-*-* 10:00:00
  Persistent=true
  Unit=zpool-scrub@%i.service
  
  [Install]
  WantedBy=timers.target

// Enable the periodic zpool scrub:
systemctl daemon-reload
systemctl enable [email protected]

// Must have for the next reboot:
vim /etc/pacman.conf
  [multilib]
  Include = /etc/pacman.d/mirrorlist
pacman -Syu networkmanager
systemctl enable NetworkManager

// (OPTIONAL) Install some extra packages:
pacman -S zsh xorg lightdm lightdm-gtk-greeter base-devel ... 

// Unmount everything before rebooting:
exit
umount /root/x/boot/efi
zpool export zroot
shutdown -h now

@Soulsuke How do you load the intel-ucode? You install the package in line 251, but I don't get it how you early load the microcode-image.
Thanks

@rakor ah, that must be a leftover from back when the guide was using zectl. With this current conf the microcode isn't used, when I'll have some free time I'll probably try to replace mkinitcpio with dracut to fix that.

EDIT: to clarify a little, this is only about early loading. Late loading is enabled by default, although it'd be better not having to rely on it

@Soulsuke from the zfsbootmenu-irc channel:

just `cat /boot/intel-ucode.img /boot/initramfs-linux.img > /boot/initramfs-linux-mc.img && ln -Tsf vmlinuz-linux /boot/vmlinuz-linux-mc`
zbm will recognize the -linux-mc pair and boot that; the kernel does the right thing when it finds two concatenated images
you should be able to wrap that in a pacman hook that will create a new pair with every kernel update
you might need to pin vmlinuz-linux-mc

@rakor Thanks for the heads up! I didn't know that was possible, I just gave it a try and works quite well :)

Added in a pacman hook to run those commands (so far it appears to be the default boot option for ZBM)

I had to use hardlinks for the microcode to work. With softlinks zbm did not show up the mc-image.

Could it be because of tue r flag? It was missing in that command you pasted, it's been working quite well so far

I've found some corner cases where the ucode hook wouldn't trigger properly.

Since maintaining hooks with explicit bash code isn't the best idea when they become too complex, I've decided to put up a repo with the scripts I use and decided to update the gist accordingly.

JFYI, there are also other ways to solve ucode problem zbm-dev/zfsbootmenu#175 (comment)

JFYI, there are also other ways to solve ucode problem zbm-dev/zfsbootmenu#175 (comment)

There are indeed other ways to accomplsh this, but require much more effort, either replacing mkinitcpio with Dracut or rebuilding ZFSBootMenu each time. Bundling the ucode sounded like the way to go

Thanks for this guide, it was quite helpful! For what it's worth, if you don't want to enter your zfs encryption password at all and don't mind storing it on a usb, I put together a small initcpio hook to load the key from usb in both Arch and in ZFSBootMenu: https://gist.github.com/branchmispredictor/779ace1349d5da503e1364a5e60d1d83

Thanks @branchmispredictor for the addition, it's surely nice to be able to use an external drive :D

On a side note, apparently there's no more need to bundle the ucode to load it early if using the microcode hook.