Fedora with Root on ZFS - installation notes

Fedora: Root on ZFS

Fedora: 28
ZFS on Linux: 0.7.9

Note: for newer Fedora releases see Reliably boot Fedora with root on ZFS.

A. Prepare source

Download Fedora Workstation network installer image.
Boot live media by adding inst.gpt at the end of the boot line and set up with:
- a boot partition /boot/efi of size 512MB
- a root partition / of size rest of the disk
- minimal installation
Reboot and:
- run dnf -y update
- set hostname, networking, idmapd
- output of rpm -qa|grep "grub2-efi-x64\|shim-x64" should contain:
```
grub2-efi-x64
 shim-x64
```
- install UEFI modules and some required packages:
```
dnf -y install grub2-efi-x64-modules kernel-devel gdisk rsync bridge-utils
```
- add system wide profile customization in /etc/profile.d/
- run init 6

Create scripts:

to run dracut and make GRUB ZFS aware:

#!/bin/bash
 kver=$1
 sh -x /usr/bin/dracut -fv --kver $kver
 mount /boot/efi
 grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
 grubby --set-default-index 0
 mkdir -p /boot/efi/EFI/fedora/x86_64-efi
 cp -a /usr/lib/grub/x86_64-efi/zfs* /boot/efi/EFI/fedora/x86_64-efi
 umount /boot/efi

to enter chroot:

#!/bin/bash
 target=$1
 mount -t proc  proc $target/proc
 mount -t sysfs sys $target/sys
 mount -o bind /dev $target/dev
 mount -o bind /dev/pts $target/dev/pts
 chroot $target /bin/env -i HOME=/root \
     TERM="$TERM" PS1='[\u@chroot \W]\$ ' \
     PATH=/bin:/usr/bin:/sbin:/usr/sbin \
     /bin/bash --login
 echo "Exiting chroot environment..."
 umount $target/dev/pts
 umount $target/dev/
 umount $target/sys/
 umount $target/proc/

make both scripts executable

Add ZFS support to GRUB in /etc/default/grub:

GRUB_PRELOADED_MODULES="zfs"

so that it can detect the ZFS filesystem; and

GRUB_DISABLE_OS_PROBER=true

so that grub2-mkconfig does not throw errors.

Fix GRUB's handling of pool device path names by creating /etc/profile.d/grub2_zpool_fix.sh containing

export ZPOOL_VDEV_NAME_PATH=YES

The effect is that zpool status will list the full path of the block device: /dev/disk/by-id/nvme0n1-part2 instead of nvme0n1-part2.

Prepare for future kernel updates:

ln -s /usr/local/sbin/zmogrify /etc/kernel/postinst.d

Add ZFS on Linux repository then install and load ZFS:

dnf install zfs zfs-dracut
modprobe zfs

Disable cache and enable scan:

systemctl disable zfs-import-cache
systemctl enable zfs-import-scan

Regenerate initramfs:

dracut -fv --kver `uname -r`

Reboot

B. Prepare target

Wipe disks:

sgdisk --zap-all /dev/disk/by-id/nvme0n1
sgdisk --zap-all /dev/disk/by-id/nvme1n1

Create boot partition (UEFI) and dedicate the rest to ZFS:

sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme0n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme0n1
sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme1n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme1n1

Format boot partition:

mkfs.fat -F32 /dev/disk/by-id/nvme0n1-part1
mkfs.fat -F32 /dev/disk/by-id/nvme1n1-part1

Create the pool:

zpool create -m none -O canmount=off -o ashift=12 -o cachefile=none rpool mirror /dev/disk/by-id/nvme0n1-part2 /dev/disk/by-id/nvme1n1-part2

Create a container dataset:

zfs create -o canmount=off -o mountpoint=none rpool/ROOT

Create the root dataset:

zfs create rpool/ROOT/default
zfs set acltype=posixacl rpool/ROOT/default # systemd-journald's /var/log/journal
zfs set xattr=sa rpool/ROOT/default
zfs set compression=on rpool/ROOT/default
zfs set atime=off rpool/ROOT/default

Create the home dataset:

zfs create -o compression=on -o atime=off -o xattr=sa -o acltype=posixacl rpool/home

Create a dataset for storing data:

zfs create -o compression=on -o atime=off -o xattr=sa -o acltype=posixacl rpool/local

Create a swap volume:

zfs create -V 16G -b $(getconf PAGESIZE) -o compression=zle -o logbias=throughput -o sync=always -o primarycache=metadata -o secondarycache=none -o com.sun:auto-snapshot=false rpool/swap
mkswap /dev/zvol/rpool/swap

Export pool:

zpool export rpool

Import pool into an alternate mount point:

zpool import rpool -d /dev/disk/by-id -o altroot=/sysroot

Set the mountpoint for the root and for home and local datasets:

zfs set mountpoint=/ rpool/ROOT/default
zfs set mountpoint=/home rpool/home # directory is empty
zfs set mountpoint=/local rpool/local # directory is empty

Check mountpoints.
Transfer data:

rsync -WSHAXhavx / /sysroot/

Copy EFI partition from the source to the target:

cd
mkdir here
mount /dev/disk/by-id/nvme0n1-part1 here
cp -a /boot/efi/* here
umount here
rmdir here

Add boot and swap partitions to fstab:

blkid /dev/disk/by-id/nvme0n1-part1

Replace content of /sysroot/etc/fstab with:

UUID=<UUID from blkid>  /boot/efi  vfat  umask=0077,shortname=winnt  0  2
/dev/zvol/rpool/swap    swap       swap  defaults                    0  0

Enter chroot:

zenter /sysroot

Needed by F28's GRUB is a line containing:

GRUB_DEVICE_BOOT=/dev/disk/by-id/nvme0n1-part2

in /etc/default/grub.

Transform target into a ZFS aware system:

zmogrify \`uname  -r\`

Confirm that GRUB can recognize the ZFS filesystem:

grub2-probe /

should output zfs.

Exit chroot
Export pool:

zpool export rpool

Reboot
Take an initial snapshot:

zfs snapshot rpool/ROOT/default@initial

Create user and group
Complete installation:

dnf -y groupinstall "Fedora Workstation"
systemctl set-default graphical.target
systemctl enable gdm.service

Reboot

References:

p7cq/Fedora_Root_On_ZFS.md

Fedora: Root on ZFS

A. Prepare source

B. Prepare target

p7cq commented Apr 26, 2020