Skip to content

Instantly share code, notes, and snippets.

@timofeika

timofeika/Arch_Linux_Root_On_ZFS.md

Forked from p7cq/Arch_Linux_Root_On_ZFS.md
Last active April 27, 2021 05:17
Show Gist options
  • Save timofeika/0992afa966c6acb392447a59185fd3c2 to your computer and use it in GitHub Desktop.
Save timofeika/0992afa966c6acb392447a59185fd3c2 to your computer and use it in GitHub Desktop.
Download ZIP
Install Arch Linux with Root on ZFS
Raw
Arch_Linux_Root_On_ZFS.md

Arch Linux Root on ZFS

Installation steps for running Arch Linux with root on ZFS using UEFI and systemd-boot. All steps are run as root.

In live environment

// Not needed

  • Set a bigger font if needed:
// setfont latarcyrheb-sun32
  • To connect to the internet add the ESSID and passphrase:
 // wpa_passphrase ESSID PASSPHRASE > /etc/wpa_supplicant/wpa_supplicant.conf
  • Start wpa_supplicant and get an IP address:
// wpa_supplicant -B -c /etc/wpa_supplicant/wpa_supplicant.conf -i <wifi interface>
// dhclient <wifi interface>

  • Wipe disks, create boot, swap and ZFS partitions:

  • Enable zfs support for arch (important)

  • curl -s https://eoli3n.github.io/archzfs/init | bash

  • Skip swap creation

  • n2:0:0 (take all free space)

  • not need 2nd disk

sgdisk --zap-all /dev/disk/by-id/<disk0>
sgdisk -n1:0:+512M -t1:ef00 /dev/disk/by-id/<disk0>
// sgdisk -n2:0:+8G -t2:8200 /dev/disk/by-id/<disk0>
sgdisk -n2:0:) -t3:bf00 /dev/disk/by-id/<disk0>

You can also use a zvol for swap, but see this first.

  • Format boot and swap partitions
mkfs.vfat /dev/disk/by-id/<disk0>-part1

mkswap /dev/disk/by-id/<disk0>-part2


- On choosing ```ashift```
// ashift=12
*You should specify an ashift when that value is too low for what you actually need, either today (disk lies) or into the future (replacement disks will be AF)*. Looks like a sound advice to me. If in doubt, [clarify](https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/) before going any further. I will go with the defaults here, which will trigger autodetection.

Disk block size check:

cat /sys/class/block//queue/{phys,log}ical_block_size


- Create pool (here's for a RAID 0 equivalent):
- Compression is zstd
- Enable autotrim

zpool create
-O atime=off
-O autotrim=on
-O acltype=posixacl -O canmount=off -O compression=zstd
-O dnodesize=legacy -O normalization=formD
-O xattr=sa -O devices=off -O mountpoint=none -R /mnt rpool /dev/disk/by-id/-part3 /dev/disk/by-id/-part3


If the pool is larger than 10 disks you should identify them ```by-path``` or ```by-vdev``` (see [here](https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#selecting-dev-names-when-creating-a-pool-linux) for more details).

Check ```ashift``` with:

zdb -C | grep ashift


- Create datasets:

zfs create -o canmount=off -o mountpoint=none rpool/ROOT zfs create -o mountpoint=/ -o canmount=noauto rpool/ROOT/default zfs create -o mountpoint=none rpool/DATA zfs create -o mountpoint=/home rpool/DATA/home zfs create -o mountpoint=/root rpool/DATA/home/root zfs create -o mountpoint=/local rpool/DATA/local zfs create -o mountpoint=none rpool/DATA/var zfs create -o mountpoint=/var/log rpool/DATA/var/log # after a rollback, systemd-journal blocks at reboot without this dataset

zpool set bootfs=rpool/ROOT/default rpool


- Create swap (not needed if you have dedicated partitions, like above)

zfs create -V 16G -b $(getconf PAGESIZE) -o compression=zle -o logbias=throughput -o sync=always -o primarycache=metadata -o secondarycache=none -o com.sun:auto-snapshot=false rpool/swap mkswap /dev/zvol/rpool/swap


- Unmount all

zfs umount -a rm -rf /mnt/*


- *Export pool*:

zpool export rpool


- Re import it:

zpool import -d /dev/disk/by-id -R /mnt rpool -N


- Mount root, then the other datasets:

zfs mount rpool/ROOT/default zfs mount -a


- Mount boot partition:

mkdir /mnt/boot mount /dev/disk/by-id/-part1 /mnt/boot


- Generate **fstab**:

mkdir /mnt/etc genfstab -U /mnt >> /mnt/etc/fstab

// not needed
- Add swap (not needed if you created swap partitions, like above):

echo "/dev/zvol/rpool/swap none swap discard 0 0" >> /mnt/etc/fstab


- Install the base system:

pacstrap /mnt base base-devel linux linux-firmware vim

If it fails, add GPG keys (see bellow).

- Change root into the new system:

arch-chroot /mnt

### In chroot

- Remove all lines in ```/etc/fstab```, leaving only the entries for ```swap``` and ```boot```.

- Add ZFS repository in ```/etc/pacman.conf```:

[archzfs] Server = http://archzfs.com/$repo/x86_64


- Add GPG keys:

curl -O https://archzfs.com/archzfs.gpg pacman-key -a archzfs.gpg pacman-key --lsign-key DDF7DB817396A49B2A2723F7403BD972F75D9D76 pacman -Syy


- Configure time zone (change accordingly):

ln -sf /usr/share/zoneinfo/Region/City /etc/localtime hwclock --systohc


- Generate locale (change accordingly):

sed -i 's/#(en_US.UTF-8)/\1/' /etc/locale.gen locale-gen echo "LANG=en_US.UTF-8" > /etc/locale.conf


- Configure **vconsole**, **hostname**, **hosts**:

echo -e "KEYMAP=us\n#FONT=latarcyrheb-sun32" > /etc/vconsole.conf echo al-zfs > /etc/hostname echo -e "127.0.0.1 localhost\n::1 localhost" >> /etc/hosts


- Set root password

- Install ZFS, microcode etc:
```bash
pacman -Syu archzfs-linux amd-ucode networkmanager sudo openssh rsync borg

I choose the default options for the archzfs-linux group: zfs-linux, zfs-utils, and mkinitcpio for initramfs.

For Intel processors install intel-ucode instead of amd-ucode.

  • Generate host id:
zgenhostid $(hostid)
  • Create cache file:
zpool set cachefile=/etc/zfs/zpool.cache rpool
  • Configure initial ramdisk in /etc/mkinitcpio.conf:
HOOKS=(base udev autodetect modconf block keyboard zfs filesystems)

and regenerate it:

mkinitcpio -p linux
  • Enable ZFS services:
systemctl enable zfs.target
systemctl enable zfs-import-cache.service
systemctl enable zfs-mount.service
systemctl enable zfs-import.target
  • Install the bootloader:
bootctl --path=/boot install
  • Add an EFI boot manager update hook in /etc/pacman.d/hooks/100-systemd-boot.hook:
[Trigger]
Type = Package
Operation = Upgrade
Target = systemd

[Action]
Description = update systemd-boot
When = PostTransaction
Exec = /usr/bin/bootctl update
  • Replace content of /boot/loader/loader.conf with:
default arch
timeout 3
# bigger boot menu on a 4K laptop display
#console-mode 1
  • Create a /boot/loader/entries/arch.conf containing:
title Arch Linux
linux /vmlinuz-linux
initrd /amd-ucode.img
initrd /initramfs-linux.img
options zfs=rpool/ROOT/default rw

If using an Intel processor, replace /amd-ucode.img with /intel-ucode.img.

  • Exit and unmount all:
exit
zfs umount -a
umount -R /mnt
  • Export pool:
zpool export rpool
  • Reboot

A minimal Arch Linux system with root on ZFS should now be configured.

Optional

  • Create user
zfs create -o mountpoint=/home/user rpool/DATA/home/user
groupadd -g 1234 group
useradd -g group -u 1234 -d /home/user -s /bin/bash user
cp /etc/skel/.bash* /home/user
chown -R user:group /home/user && chmod 700 /home/user
  • Create non-root pools:
zpool create \
    -O atime=off \
    -O acltype=posixacl -O canmount=off -O compression=lz4 \
    -O dnodesize=auto \
    -O xattr=sa -O devices=off -O mountpoint=none pool:a /dev/disk/by-id/<disk2> /dev/disk/by-id/<disk3>

zpool create \
    -O atime=off \
    -O acltype=posixacl -O canmount=off -O compression=lz4 \
    -O dnodesize=auto \
    -O xattr=sa -O devices=off -O mountpoint=none pool:b mirror /dev/disk/by-id/<disk4> /dev/disk/by-id/<disk5> mirror /dev/disk/by-id/<disk6> /dev/disk/by-id/<disk7>
  • Create non-root datasets:
zfs create -o canmount=off -o mountpoint=none pool:a/DATA
zfs create -o mountpoint=/path pool:a/DATA/path
zfs create -o mountpoint=/path/games -o recordsize=1M pool:a/DATA/path/games
zfs create -o mountpoint=/path/transmission -o recordsize=1M pool:a/DATA/path/transmission
zfs create -o mountpoint=/path/backup -o compression=off pool:a/DATA/path/backup
  • Create NFS share:

Set Domain in idmapd.conf on server and clients.

zfs set [email protected]/24 pool:a/DATA/path/name
systemctl enable nfs-server.service zfs-share.service --now

References:

  1. https://wiki.archlinux.org/index.php/Install_Arch_Linux_on_ZFS
  2. https://wiki.archlinux.org/index.php/ZFS
  3. https://ramsdenj.com/2016/06/23/arch-linux-on-zfs-part-2-installation.html
  4. https://github.com/reconquest/archiso-zfs
  5. https://zedenv.readthedocs.io/en/latest/setup.html
  6. https://docs.oracle.com/cd/E37838_01/html/E60980/index.html
  7. https://ramsdenj.com/2020/03/18/zectl-zfs-boot-environment-manager-for-linux.html
  8. https://superuser.com/questions/1310927/what-is-the-absolute-minimum-size-a-uefi-partition-can-be, https://systemd.io/9OOT_LOADER_SPECIFICATION/
  9. OpenZFS Admin Documentation
  10. zfs(8)
  11. zpool(8)
  12. https://jrs-s.net/category/open-source/zfs/
  13. https://github.com/ewwhite/zfs-ha/wiki
  14. http://nex7.blogspot.com/2013/03/readme1st.html
Raw
zsrs
#!/usr/bin/env bash
#
# zsrs, a ZFS snapshot/rollback script
#
# @p7cq
ss_to_keep=0
bootmark="BOOT"
root_pool="rpool/ROOT/default"
root_ss="/.zfs/snapshot"
boot_ss="/local/.boot/snapshot/${bootmark}"
boot_part="/boot"
loader_file="${boot_part}/loader/entries/arch.conf"
action=${1}
ss_date=$(/usr/bin/date +"%A %B %e, %Y %k:%M:%S")
ss_manifest_file=".manifest"
target_dataset=${root_pool}
mount_name="snapshot"
function _help() {
local topic=${1}
[[ ${topic} == "usage" ]] && \
/usr/bin/echo -e "usage:\n \
zsrs delete [snapshot id] remove all snapshots keeping only the last ${ss_to_keep}\n \
zsrs snapshot [comment] snapshot the system; comment max length 64 chars\n \
zsrs rollback [snapshot id] rollback the system to the previous state (the last snapshot)\n \
zsrs mount <snapshot id> clone snapshot\n \
zsrs umount <snapshot id> destroy clone\n \
zsrs list list snapshots\n \
zsrs help usage" && \
exit 0
}
function _boot_ss() {
[[ -d "${1}" ]] && /usr/bin/echo "\e[97m\e[1mboot snapshot id is undefined\e[0m" && exit 1
local ss_id="${1}"
/usr/bin/mkdir ${boot_ss}/${ss_id}
/usr/bin/echo ${ss_manifest} > ${boot_ss}/${ss_id}/${ss_manifest_file}
/usr/bin/rsync -avzq --delete ${boot_part}/ ${boot_ss}/${ss_id}/${bootmark}
}
function _boot_rb() {
local ss_id="${1}"
[[ -z "${ss_id}" ]] && /usr/bin/echo "\e[97m\e[1mboot snapshot id is undefined\e[0m" && exit 1
/usr/bin/echo -e "\e[97m\e[1moverwriting boot partition at \e[0m\e[33m\e[1m${boot_part}\e[0m"
/usr/bin/rsync -avzq --delete ${boot_ss}/${ss_id}/${bootmark}/ ${boot_part}
IFS='^' read -ra manifest <<< "$(/usr/bin/cat ${boot_ss}/${ss_id}/${ss_manifest_file})"
local title="title Arch Linux (${ss_id}: ${manifest[0]} ${manifest[1]})"
/usr/bin/sed -i "1 s/^.*$/${title}/g" ${loader_file}
}
function _ss_manifest() {
local comment="${1}"
ss_manifest="^${ss_date}"
[[ -z "${comment}" ]] && comment="snapshot"
comment=$(/usr/bin/sed 's/\^/_/g' <<< ${comment})
ss_manifest="${comment:0:64}${ss_manifest}"
}
function _restrict() {
if (( ${EUID:-0} || "$(/usr/bin/id -u)" )); then
/usr/bin/echo -e "\e[97m\e[1myou must be root to do this\e[0m"
exit 1
fi
}
declare -a args=("delete" "d" "snapshot" "s" "rollback" "r" "list" "l" "help" "h" "mount" "m" "umount" "u")
[[ "$#" = "0" ]] || [[ ! " ${args[@]} " =~ " ${1} " ]] && _help "usage" && exit 0
[[ ! -d ${root_ss} ]] && /usr/bin/echo "${root_ss} does not exist" && exit 1
[[ ! -d ${boot_ss} ]] && /usr/bin/echo "${boot_ss} does not exist" && exit 1
ss_count=$(/usr/bin/ls ${boot_ss} | /usr/bin/wc -l)
if [[ ${action} == "delete" ]] || [[ ${action} == "d" ]]; then
_restrict
ss_id=${2}
if (( $# == 1 )); then
/usr/bin/echo -en "\e[97m\e[1mdelete all and keep the last \e[0m\e[33m\e[1m${ss_to_keep}\e[0m\e[97m\e[1m snapshots? (yes/no): \e[0m" && read
[[ "${REPLY}" != "yes" ]] && exit 0
fi
if [[ ! -z "${ss_id}" ]]; then
re='^[0-9]+$'
[[ ! ${ss_id} =~ ${re} ]] && /usr/bin/echo -e "\e[97m\e[1ma positive integer is expected (see \e[0m\e[33m\e[1mzsrs list\e[0m)\e[0m" && exit 1
[[ ! -d "${root_ss}/${ss_id}" ]] && /usr/bin/echo -e "\e[97m\e[1mno such snapshot\e[0m" && exit 1
/usr/bin/zfs destroy ${root_pool}@${ss_id}
/usr/bin/rm -rf ${boot_ss}/${ss_id}
exit 0
fi
if (( ss_count > ss_to_keep )); then
lwm=$(/usr/bin/ls ${root_ss} | /usr/bin/sort -n | /usr/bin/head -1)
hwm=$(/usr/bin/ls ${root_ss} | /usr/bin/sort -nr | /usr/bin/head -1)
end=$((hwm-ss_to_keep))
if [ ! -z "${lwm}" ] && [ ! -z "${hwm}" ]; then
for ((s=lwm; s<=end; s++)) do
[[ ! -d "${boot_ss}/${s}" ]] && continue
/usr/bin/zfs destroy ${root_pool}@${s}
/usr/bin/rm -rf ${boot_ss}/${s}
done
fi
else
/usr/bin/echo -e "\e[97m\e[1mless than \e[0m\e[33m\e[1m$((${ss_to_keep}+1))\e[0m\e[97m\e[1m snapshots, none deleted\e[0m"
fi
exit 0
fi
if [[ ${action} == "snapshot" ]] || [[ ${action} == "s" ]]; then
_restrict
ss_id=$(($(/usr/bin/ls ${boot_ss} | /usr/bin/sort -nr | /usr/bin/head -1)+1))
_ss_manifest "${2}"
_boot_ss ${ss_id}
/usr/bin/zfs snapshot ${root_pool}@${ss_id}
/usr/bin/echo -e "\e[97m\e[1mcreated snapshot with id \e[0m\e[33m\e[1m${ss_id}\e[0m"
exit 0
fi
if [[ ${action} == "rollback" ]] || [[ ${action} == "r" ]]; then
_restrict
ss_id=${2}
last_ss=$(/usr/bin/ls ${boot_ss} | /usr/bin/sort -nr | /usr/bin/head -1)
if [[ ! -z "${ss_id}" ]]; then
([[ "${ss_id}" == "0" ]] || [[ ! -d "${boot_ss}/${ss_id}" ]] || [[ ! -d "${root_ss}/${ss_id}" ]]) && /usr/bin/echo -e "\e[97m\e[1mno such snapshot\e[0m" && exit 1
re='^[0-9]+$'
[[ ! ${ss_id} =~ ${re} ]] && /usr/bin/echo -e "\e[97m\e[1ma positive integer is expected\e[0m" && exit 1
else
ss_id=${last_ss}
fi
IFS='^' read -ra manifest <<< "$(/usr/bin/cat ${boot_ss}/${ss_id}/${ss_manifest_file})"
/usr/bin/echo -e "\e[97m\e[1mrolling back to snapshot \e[0m\e[33m\e[1m${ss_id}\e[0m\e[97m\e[1m: \e[0m\e[33m\e[1m${manifest[0]}\e[97m\e[1m taken on \e[0m\e[33m\e[1m${manifest[1]}\e[0m"
/usr/bin/echo -en "\e[97m\e[1ma reboot will be required, proceed? (yes/no): \e[0m" && read
[[ "${REPLY}" != "yes" ]] && exit 0
destroy_recent=""
if (( ss_id < last_ss )); then
read -p $'\e[97m\e[1mmore recent snapshots exist, delete them? (yes/no): \e[0m' answer
[[ "${answer}" != "yes" ]] && /usr/bin/echo -e "\e[97m\e[1mexiting...\e[0m" && exit 0
destroy_recent="-r"
for ((s=last_ss; s>ss_id; s--)); do
more_recent_ss="${boot_ss}/${s}"
[[ -d "${more_recent_ss}" ]] && /usr/bin/rm -rf ${more_recent_ss}
done
fi
/usr/bin/zfs rollback ${destroy_recent} ${root_pool}@${ss_id}
_boot_rb ${ss_id}
/usr/bin/echo -e "\e[97m\e[1msnapshot \e[0m\e[33m\e[1m${manifest[0]}\e[0m\e[97m\e[1m is now the active system, rebooting...\e[0m"
sleep 2
/usr/bin/systemctl reboot
fi
if [[ ${action} == "mount" ]] || [[ ${action} == "m" ]]; then
_restrict
ss_id=${2}
re='^[0-9]+$'
[[ ! ${ss_id} =~ ${re} ]] && /usr/bin/echo -e "\e[97m\e[1ma positive integer is expected\e[0m" && exit 1
([[ "${ss_id}" == "0" ]] || [[ ! -d "${boot_ss}/${ss_id}" ]] || [[ ! -d "${root_ss}/${ss_id}" ]]) && /usr/bin/echo -e "\e[97m\e[1mno such snapshot\e[0m" && exit 1
mountpoint=$(/usr/bin/zfs get mountpoint -o value ${target_dataset} | /usr/bin/sed -n 2p)
filesystem=${mount_name}:${ss_id}
[[ -d "${mountpoint}${filesystem}" ]] && /usr/bin/echo -e "\e[97m\e[1msnapshot \e[0m\e[33m\e[1m${ss_id}\e[0m\e[97m\e[1m is already mounted at \e[33m\e[1m${mountpoint}${filesystem}\e[0m" && exit 1
/usr/bin/zfs clone ${root_pool}@${ss_id} ${target_dataset}/${filesystem}
/usr/bin/echo -e "\e[97m\e[1msnapshot \e[0m\e[33m\e[1m${ss_id}\e[0m\e[97m\e[1m available at \e[33m\e[1m${mountpoint}${filesystem}\e[0m"
fi
if [[ ${action} == "umount" ]] || [[ ${action} == "u" ]]; then
_restrict
ss_id=${2}
re='^[0-9]+$'
[[ ! ${ss_id} =~ ${re} ]] && /usr/bin/echo -e "\e[97m\e[1ma positive integer is expected\e[0m" && exit 1
([[ "${ss_id}" == "0" ]] || [[ ! -d "${boot_ss}/${ss_id}" ]] || [[ ! -d "${root_ss}/${ss_id}" ]]) && /usr/bin/echo -e "\e[97m\e[1mno such snapshot\e[0m" && exit 1
mountpoint=$(/usr/bin/zfs get mountpoint -o value ${target_dataset} | /usr/bin/sed -n 2p)
filesystem=${mount_name}:${ss_id}
[[ ! -d "${mountpoint}${filesystem}" ]] && /usr/bin/echo -e "\e[97m\e[1msnapshot \e[0m\e[33m\e[1m${ss_id}\e[0m\e[97m\e[1m is not mounted\e[0m" && exit 1
/usr/bin/zfs destroy ${target_dataset}/${filesystem}
/usr/bin/echo -e "\e[97m\e[1msnapshot \e[0m\e[33m\e[1m${ss_id}\e[0m\e[97m\e[1m unmounted\e[0m"
fi
if [[ ${action} == "help" ]] || [[ ${action} == "h" ]]; then
_help "usage"
exit 0
fi
if [[ ${action} == "list" ]] || [[ ${action} == "l" ]]; then
if ((ss_count == 0)); then
/usr/bin/echo -e "\e[97m\e[1mno snapshots available\e[0m"
exit 0
fi
for id in $(/usr/bin/ls ${boot_ss} | /usr/bin/sort -nr); do
IFS='^' read -ra manifest <<< "$(/usr/bin/cat ${boot_ss}/${id}/${ss_manifest_file})"
/usr/bin/echo -e "\e[33m\e[1m${id}\e[0m^\e[97m\e[1m${manifest[0]}^${manifest[1]}\e[0m"
done | /usr/bin/column -t -N ID,COMMENT,DATE -s "^"
exit 0
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment