Fedora with Root on ZFS - installation notes

Fedora_Root_On_ZFS.md

Fedora: Root on ZFS

• Fedora: 28
• ZFS on Linux: 0.7.9

---

Note: for newer Fedora releases see Reliably boot Fedora with root on ZFS.

A. Prepare source

1. Download Fedora Workstation network installer image.

2. Boot live media by adding inst.gpt at the end of the boot line and set up with:

   • a boot partition /boot/efi of size 512MB
   • a root partition / of size rest of the disk
   • minimal installation

3. Reboot and:

   • run dnf -y update
   • set hostname, networking, idmapd
   • output of rpm -qa|grep "grub2-efi-x64\|shim-x64" should contain:

   grub2-efi-x64
    shim-x64

   • install UEFI modules and some required packages:

   dnf -y install grub2-efi-x64-modules kernel-devel gdisk rsync bridge-utils

   • add system wide profile customization in /etc/profile.d/
   • run init 6

4. Create scripts:

   • to run dracut and make GRUB ZFS aware:

   #!/bin/bash
    kver=$1
    sh -x /usr/bin/dracut -fv --kver $kver
    mount /boot/efi
    grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
    grubby --set-default-index 0
    mkdir -p /boot/efi/EFI/fedora/x86_64-efi
    cp -a /usr/lib/grub/x86_64-efi/zfs* /boot/efi/EFI/fedora/x86_64-efi
    umount /boot/efi

   • to enter chroot:

   #!/bin/bash
    target=$1
    mount -t proc  proc $target/proc
    mount -t sysfs sys $target/sys
    mount -o bind /dev $target/dev
    mount -o bind /dev/pts $target/dev/pts
    chroot $target /bin/env -i HOME=/root \
        TERM="$TERM" PS1='[\u@chroot \W]\$ ' \
        PATH=/bin:/usr/bin:/sbin:/usr/sbin \
        /bin/bash --login
    echo "Exiting chroot environment..."
    umount $target/dev/pts
    umount $target/dev/
    umount $target/sys/
    umount $target/proc/

   • make both scripts executable

5. Add ZFS support to GRUB in /etc/default/grub:

GRUB_PRELOADED_MODULES="zfs"

so that it can detect the ZFS filesystem; and

GRUB_DISABLE_OS_PROBER=true

so that grub2-mkconfig does not throw errors.

6. Fix GRUB's handling of pool device path names by creating /etc/profile.d/grub2_zpool_fix.sh containing

export ZPOOL_VDEV_NAME_PATH=YES

The effect is that zpool status will list the full path of the block device: /dev/disk/by-id/nvme0n1-part2 instead of nvme0n1-part2.

7. Prepare for future kernel updates:

ln -s /usr/local/sbin/zmogrify /etc/kernel/postinst.d

8. Add ZFS on Linux repository then install and load ZFS:

dnf install zfs zfs-dracut
modprobe zfs

9. Disable cache and enable scan:

systemctl disable zfs-import-cache
systemctl enable zfs-import-scan

10. Regenerate initramfs:

dracut -fv --kver `uname -r`

11. Reboot

---

B. Prepare target

1. Wipe disks:

sgdisk --zap-all /dev/disk/by-id/nvme0n1
sgdisk --zap-all /dev/disk/by-id/nvme1n1

2. Create boot partition (UEFI) and dedicate the rest to ZFS:

sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme0n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme0n1
sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme1n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme1n1

3. Format boot partition:

mkfs.fat -F32 /dev/disk/by-id/nvme0n1-part1
mkfs.fat -F32 /dev/disk/by-id/nvme1n1-part1

4. Create the pool:

zpool create -m none -O canmount=off -o ashift=12 -o cachefile=none rpool mirror /dev/disk/by-id/nvme0n1-part2 /dev/disk/by-id/nvme1n1-part2

5. Create a container dataset:

zfs create -o canmount=off -o mountpoint=none rpool/ROOT

6. Create the root dataset:

zfs create rpool/ROOT/default
zfs set acltype=posixacl rpool/ROOT/default # systemd-journald's /var/log/journal
zfs set xattr=sa rpool/ROOT/default
zfs set compression=on rpool/ROOT/default
zfs set atime=off rpool/ROOT/default

7. Create the home dataset:

zfs create -o compression=on -o atime=off -o xattr=sa -o acltype=posixacl rpool/home

8. Create a dataset for storing data:

zfs create -o compression=on -o atime=off -o xattr=sa -o acltype=posixacl rpool/local

9. Create a swap volume:

zfs create -V 16G -b $(getconf PAGESIZE) -o compression=zle -o logbias=throughput -o sync=always -o primarycache=metadata -o secondarycache=none -o com.sun:auto-snapshot=false rpool/swap
mkswap /dev/zvol/rpool/swap

10. Export pool:

zpool export rpool

11. Import pool into an alternate mount point:

zpool import rpool -d /dev/disk/by-id -o altroot=/sysroot

12. Set the mountpoint for the root and for home and local datasets:

zfs set mountpoint=/ rpool/ROOT/default
zfs set mountpoint=/home rpool/home # directory is empty
zfs set mountpoint=/local rpool/local # directory is empty

13. Check mountpoints.

14. Transfer data:

rsync -WSHAXhavx / /sysroot/

15. Copy EFI partition from the source to the target:

cd
mkdir here
mount /dev/disk/by-id/nvme0n1-part1 here
cp -a /boot/efi/* here
umount here
rmdir here

15. Add boot and swap partitions to fstab:

blkid /dev/disk/by-id/nvme0n1-part1

Replace content of /sysroot/etc/fstab with:

UUID=<UUID from blkid>  /boot/efi  vfat  umask=0077,shortname=winnt  0  2
/dev/zvol/rpool/swap    swap       swap  defaults                    0  0

17. Enter chroot:

zenter /sysroot

18. Needed by F28's GRUB is a line containing:

GRUB_DEVICE_BOOT=/dev/disk/by-id/nvme0n1-part2

in /etc/default/grub.

19. Transform target into a ZFS aware system:

zmogrify \`uname  -r\`

20. Confirm that GRUB can recognize the ZFS filesystem:

grub2-probe /

should output zfs.

21. Exit chroot

22. Export pool:

zpool export rpool

23. Reboot

24. Take an initial snapshot:

zfs snapshot rpool/ROOT/default@initial

25. Create user and group

26. Complete installation:

dnf -y groupinstall "Fedora Workstation"
systemctl set-default graphical.target
systemctl enable gdm.service

27. Reboot

---

References:

• https://www.csparks.com/BootFedoraZFS/index.md
• http://matthewheadlee.com/projects/zfs-as-root-filesystem-on-fedora
• https://ramsdenj.com/2016/06/23/arch-linux-on-zfs-part-2-installation.html
• https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS
• https://wiki.archlinux.org/index.php/ZFS
• https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-use-a-zvol-as-a-swap-device
• https://illumos.org/man/1M/zpool
• https://illumos.org/man/1M/zfs
• https://docs.fedoraproject.org/en-US/fedora/f28/install-guide/advanced/Boot_Options

If you change your sgdisk commands to be like this you won't have to 'fix' the partition numbering:

sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme0n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme0n1
sgdisk -n1:1M:+512M -t1:EF00 /dev/disk/by-id/nvme1n1
sgdisk -n2:0:0 -t2:BF01 /dev/disk/by-id/nvme1n1

Indeed, thank you,