-
-
Save kiela/6012350 to your computer and use it in GitHub Desktop.
HOWTO: Hetzner + FreeBSD 13.1 + mirrored root on ZFS + optimized for 4K drives
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Wipe out disks from whatever they contain. | |
| root@mfsbsd:~ # dd if=/dev/zero of=/dev/nvd0 bs=1M status=progress | |
| dd: /dev/nvd0: short write on character devicesferred 1186.001s, 3237 MB/s | |
| dd: /dev/nvd0: end of device | |
| 3662831+0 records in | |
| 3662830+1 records out | |
| 3840755982336 bytes transferred in 1186.555877 secs (3236894322 bytes/sec) | |
| root@mfsbsd:~ # dd if=/dev/zero of=/dev/nvd1 bs=1M status=progress | |
| dd: /dev/nvd1: short write on character devicesferred 1187.004s, 3233 MB/s | |
| dd: /dev/nvd1: end of device | |
| 3662831+0 records in | |
| 3662830+1 records out | |
| 3840755982336 bytes transferred in 1187.841487 secs (3233391009 bytes/sec) | |
| root@mfsbsd:~ # kldload zfs | |
| root@mfsbsd:~ # sysctl kern.geom.label.gptid.enable=0 | |
| kern.geom.label.gptid.enable: 1 -> 0 | |
| root@mfsbsd:~ # sysctl kern.geom.label.disk_ident.enable=0 | |
| kern.geom.label.disk_ident.enable: 1 -> 0 | |
| root@mfsbsd:~ # gpart destroy -F nvd0 | |
| nvd0 destroyed | |
| root@mfsbsd:~ # gpart destroy -F nvd1 | |
| nvd1 destroyed | |
| # Getting "gpart: arg0 'nvdN': Invalid argument" is alright as it means there was no partition table on the disk anyway. | |
| root@mfsbsd:~ # zpool labelclear -f /dev/nvd0 | |
| root@mfsbsd:~ # zpool labelclear -f /dev/nvd1 | |
| root@mfsbsd:~ # gpart create -s gpt nvd0 | |
| nvd0 created | |
| root@mfsbsd:~ # gpart create -s gpt nvd1 | |
| nvd1 created | |
| # Set minimum sector size. The default value is 9 which represents a 512-bytes sector (2^9 = 512) while 12 represents 4096-bytes (or 4k) sector (2^12 = 4096). | |
| # Before setting anything, make sure your disks can use 4096-bytes sectors and announce that to the OS. | |
| # To get information about your disks you can use diskinfo. The output shows sectorsize which is the size the I/O system sees and stripesize is the actual hardware block size used by the disk. | |
| root@mfsbsd:~ # diskinfo -v nvd0 | grep -E "sectorsize|stripesize" | |
| 512 # sectorsize | |
| 4096 # stripesize | |
| root@mfsbsd:~ # diskinfo -v nvd0 | grep -E "sectorsize|stripesize" | |
| 512 # sectorsize | |
| 4096 # stripesize | |
| # NOTE: SSD disks are 4k from the beginning so if you use SSD disks, you better force it. | |
| root@mfsbsd:~ # sysctl vfs.zfs.min_auto_ashift=12 | |
| vfs.zfs.min_auto_ashift: 9 -> 12 | |
| # Boot partition for ZFS on UEFI. | |
| root@mfsbsd:~ # gpart add -a 4k -s 260M -t efi -l efiboot0 nvd0 | |
| nvd0p1 added | |
| root@mfsbsd:~ # gpart add -a 4k -s 260M -t efi -l efiboot1 nvd1 | |
| nvd1p1 added | |
| # Boot partition for ZFS on a legacy platforms. | |
| root@mfsbsd:~ # gpart add -a 4k -s 512K -t freebsd-boot -l gptboot0 nvd0 | |
| nvd0p2 added | |
| root@mfsbsd:~ # gpart add -a 4k -s 512K -t freebsd-boot -l gptboot1 nvd1 | |
| nvd1p2 added | |
| # Last partition for everything else. Starting from the next 4096-bytes sector. | |
| root@mfsbsd:~ # gpart add -a 4k -b 536576 -t freebsd-zfs -l zroot0 nvd0 | |
| nvd0p3 added | |
| root@mfsbsd:~ # gpart add -a 4k -b 536576 -t freebsd-zfs -l zroot1 nvd1 | |
| nvd1p3 added | |
| # Clearing zpool labels at this point again is something that I noticed in bsdinstall so it's just for bsdinstall compatibility. | |
| root@mfsbsd:~ # zpool labelclear -f /dev/nvd0p3 | |
| failed to clear label for /dev/nvd0p3 | |
| root@mfsbsd:~ # zpool labelclear -f /dev/nvd1p3 | |
| failed to clear label for /dev/nvd1p3 | |
| # If you want to check how disk was partitioned, you can check that with gpart show: | |
| root@mfsbsd:~ # gpart show nvd0 nvd1 | |
| => 40 7501476448 nvd0 GPT (3.5T) | |
| 40 532480 1 efi (260M) | |
| 532520 1024 2 freebsd-boot (512K) | |
| 533544 3032 - free - (1.5M) | |
| 536576 7500939912 3 freebsd-zfs (3.5T) | |
| => 40 7501476448 nvd1 GPT (3.5T) | |
| 40 532480 1 efi (260M) | |
| 532520 1024 2 freebsd-boot (512K) | |
| 533544 3032 - free - (1.5M) | |
| 536576 7500939912 3 freebsd-zfs (3.5T) | |
| root@mfsbsd:~ # mdconfig -a -t malloc -s 128m -u 2 | |
| root@mfsbsd:~ # newfs -O2 /dev/md2 | |
| /dev/md2: 128.0MB (262144 sectors) block size 32768, fragment size 4096 | |
| using 4 cylinder groups of 32.03MB, 1025 blks, 4224 inodes. | |
| super-block backups (for fsck_ffs -b #) at: | |
| 192, 65792, 131392, 196992 | |
| root@mfsbsd:~ # mount /dev/md2 /boot/zfs | |
| root@mfsbsd:~ # zpool create -m none zroot mirror gpt/zroot0 gpt/zroot1 | |
| # If you want to check if it is 4K (2^12=4096) and not 512 (2^9=512) just run: | |
| root@mfsbsd:~ # zdb | grep ashift | |
| ashift: 12 | |
| root@mfsbsd:~ # zpool status | |
| pool: zroot | |
| state: ONLINE | |
| config: | |
| NAME STATE READ WRITE CKSUM | |
| zroot ONLINE 0 0 0 | |
| mirror-0 ONLINE 0 0 0 | |
| gpt/zroot0 ONLINE 0 0 0 | |
| gpt/zroot1 ONLINE 0 0 0 | |
| errors: No known data errors | |
| # Disable mounting zroot pool itself. We are going to create and use zroot/ROOT as dedicated dateset. | |
| root@mfsbsd:~ # zfs set canmount=off zroot | |
| # Set checksum mechanism to use fletcher4. If you are paranoid, you can change it to sha256. I don't like being paranoid. | |
| root@mfsbsd:~ # zfs set checksum=fletcher4 zroot | |
| # Turn off writing a metadata change every time a file is accessed to increase performance. | |
| root@mfsbsd:~ # zfs set atime=off zroot | |
| root@mfsbsd:~ # zfs create -o canmount=off -o mountpoint=none zroot/ROOT | |
| root@mfsbsd:~ # zfs create -o reservation=10G -o mountpoint=/boot/zfs/zroot zroot/ROOT/default | |
| root@mfsbsd:~ # zfs create -o compression=on -o exec=on -o setuid=off -o mountpoint=/boot/zfs/zroot/tmp zroot/tmp | |
| root@mfsbsd:~ # zfs create -o reservation=10G -o mountpoint=/boot/zfs/zroot/home zroot/home | |
| root@mfsbsd:~ # zfs create -o reservation=10G -o mountpoint=/boot/zfs/zroot/usr zroot/usr | |
| root@mfsbsd:~ # zfs create -o mountpoint=/boot/zfs/zroot/usr/local zroot/usr/local | |
| root@mfsbsd:~ # zfs create -o compression=lz4 -o setuid=off -o mountpoint=/boot/zfs/zroot/usr/ports zroot/usr/ports | |
| root@mfsbsd:~ # zfs create -o compression=off -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/usr/ports/distfiles zroot/usr/ports/distfiles | |
| root@mfsbsd:~ # zfs create -o compression=off -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/usr/ports/packages zroot/usr/ports/packages | |
| root@mfsbsd:~ # zfs create -o compression=lz4 -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/usr/src zroot/usr/src | |
| root@mfsbsd:~ # zfs create -o reservation=10G -o mountpoint=/boot/zfs/zroot/var zroot/var | |
| root@mfsbsd:~ # zfs create -o compression=lz4 -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/var/crash zroot/var/crash | |
| root@mfsbsd:~ # zfs create -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/var/db zroot/var/db | |
| root@mfsbsd:~ # zfs create -o compression=lz4 -o exec=on -o setuid=off -o mountpoint=/boot/zfs/zroot/var/db/pkg zroot/var/db/pkg | |
| root@mfsbsd:~ # zfs create -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/var/empty zroot/var/empty | |
| root@mfsbsd:~ # zfs create -o compression=lz4 -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/var/log zroot/var/log | |
| root@mfsbsd:~ # zfs create -o compression=gzip -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/var/mail zroot/var/mail | |
| root@mfsbsd:~ # zfs create -o exec=off -o setuid=off -o mountpoint=/boot/zfs/zroot/var/run zroot/var/run | |
| root@mfsbsd:~ # zfs create -o compression=lz4 -o exec=on -o setuid=off -o mountpoint=/boot/zfs/zroot/var/tmp zroot/var/tmp | |
| root@mfsbsd:~ # zfs create -V 16G -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none zroot/swap | |
| root@mfsbsd:~ # chmod 1777 /boot/zfs/zroot/tmp | |
| root@mfsbsd:~ # chmod 1777 /boot/zfs/zroot/var/tmp | |
| # Install FreeBSD. | |
| root@mfsbsd:~ # cd /boot/zfs/zroot/ | |
| root@mfsbsd:/boot/zfs/zroot # fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/13.1/FreeBSD-13.1-RELEASE-amd64-disc1.iso | |
| FreeBSD-13.1-RELEASE-amd64-disc1.iso 998 MB 41 MBps 24s | |
| root@mfsbsd:/boot/zfs/zroot # mkdir /cdrom | |
| root@mfsbsd:/boot/zfs/zroot # mount -t cd9660 /dev/`mdconfig -f FreeBSD-13.1-RELEASE-amd64-disc1.iso` /cdrom | |
| root@mfsbsd:/boot/zfs/zroot # unxz -c /cdrom/usr/freebsd-dist/base.txz | tar xpf - | |
| root@mfsbsd:/boot/zfs/zroot # unxz -c /cdrom/usr/freebsd-dist/kernel.txz | tar xpf - | |
| root@mfsbsd:/boot/zfs/zroot # unxz -c /cdrom/usr/freebsd-dist/lib32.txz | tar xpf - | |
| root@mfsbsd:/boot/zfs/zroot # unxz -c /cdrom/usr/freebsd-dist/ports.txz | tar xpf - | |
| root@mfsbsd:/boot/zfs/zroot # unxz -c /cdrom/usr/freebsd-dist/src.txz | tar xpf - | |
| root@mfsbsd:/boot/zfs/zroot # umount /cdrom | |
| root@mfsbsd:/boot/zfs/zroot # rm FreeBSD-13.1-RELEASE-amd64-disc1.iso | |
| root@mfsbsd:/boot/zfs/zroot # zfs set readonly=on zroot/var/empty | |
| root@mfsbsd:/boot/zfs/zroot # chroot /boot/zfs/zroot /bin/csh | |
| root@mfsbsd:/ # sysrc hostname="jarl.eilie.io" | |
| root@mfsbsd:/ # sysrc ifconfig_igb0="DHCP" | |
| root@mfsbsd:/ # sysrc ifconfig_igb0_ipv6="inet6 accept_rtadv" | |
| root@mfsbsd:/ # sysrc zfs_enable="YES" | |
| root@mfsbsd:/ # sysrc sshd_enable="YES" | |
| root@mfsbsd:/ # sysrc dumpdev="AUTO" | |
| root@mfsbsd:/ # sysrc clear_tmp_enable="YES" | |
| root@mfsbsd:/ # sysrc syslogd_flags="-ss" | |
| root@mfsbsd:/ # sysrc sendmail_enable="NONE" | |
| root@mfsbsd:/ # touch /etc/fstab | |
| root@mfsbsd:/ # cat <<EOF > /boot/loader.conf | |
| kern.geom.label.gptid.enable="0" | |
| kern.geom.label.disk_ident.enable="0" | |
| vfs.zfs.min_auto_ashift=12 | |
| vfs.zfs.prefetch_disable="1" | |
| zfs_load="YES" | |
| vfs.root.mountfrom="zfs:zroot/ROOT/default" | |
| # Turn off ACPI errors | |
| debug.acpi.disabled="thermal" | |
| EOF | |
| root@mfsbsd:/ # tzsetup | |
| root@mfsbsd:/ # kbdmap | |
| root@mfsbsd:/ # passwd root | |
| root@mfsbsd:/ # adduser | |
| root@mfsbsd:/ # cd /etc/mail && make aliases | |
| root@mfsbsd:/etc/mail # exit | |
| root@mfsbsd:/boot/zfs/zroot # cd | |
| root@mfsbsd:~ # cp /etc/zfs/zpool.cache /boot/zfs/zroot/etc/zfs/zpool.cache | |
| # For ZFS on UEFI. | |
| root@mfsbsd:~ # newfs_msdos -F 32 -c 1 /dev/nvd0p1 | |
| /dev/nvd0p1: 524256 sectors in 524256 FAT32 clusters (512 bytes/cluster) | |
| BytesPerSec=512 SecPerClust=1 ResSectors=32 FATs=2 Media=0xf0 SecPerTrack=63 Heads=255 HiddenSecs=0 HugeSectors=532480 FATsecs=4096 RootCluster=2 FSInfo=1 Backup=2 | |
| root@mfsbsd:~ # mount -t msdosfs /dev/nvd0p1 /mnt | |
| root@mfsbsd:~ # mkdir -p /mnt/efi/{boot,freebsd} | |
| root@mfsbsd:~ # cp /boot/zfs/zroot/boot/loader.efi /mnt/efi/boot/bootx64.efi | |
| root@mfsbsd:~ # cp /boot/zfs/zroot/boot/loader.efi /mnt/efi/freebsd/loader.efi | |
| root@mfsbsd:~ # umount /mnt | |
| root@mfsbsd:~ # newfs_msdos -F 32 -c 1 /dev/nvd1p1 | |
| /dev/nvd1p1: 524256 sectors in 524256 FAT32 clusters (512 bytes/cluster) | |
| BytesPerSec=512 SecPerClust=1 ResSectors=32 FATs=2 Media=0xf0 SecPerTrack=63 Heads=255 HiddenSecs=0 HugeSectors=532480 FATsecs=4096 RootCluster=2 FSInfo=1 Backup=2 | |
| root@mfsbsd:~ # mount -t msdosfs /dev/nvd1p1 /mnt | |
| root@mfsbsd:~ # mkdir -p /mnt/efi/{boot,freebsd} | |
| root@mfsbsd:~ # cp /boot/zfs/zroot/boot/loader.efi /mnt/efi/boot/bootx64.efi | |
| root@mfsbsd:~ # cp /boot/zfs/zroot/boot/loader.efi /mnt/efi/freebsd/loader.efi | |
| root@mfsbsd:~ # umount /mnt | |
| # For ZFS on a legacy platforms. | |
| root@mfsbsd:~ # gpart bootcode -b /boot/zfs/zroot/boot/pmbr -p /boot/zfs/zroot/boot/gptzfsboot -i 2 nvd0 | |
| partcode written to nvd0p2 | |
| bootcode written to nvd0 | |
| root@mfsbsd:~ # gpart bootcode -b /boot/zfs/zroot/boot/pmbr -p /boot/zfs/zroot/boot/gptzfsboot -i 2 nvd1 | |
| partcode written to nvd1p2 | |
| bootcode written to nvd1 | |
| # Prepare ZFS before rebooting the system. | |
| root@mfsbsd:~ # zfs unmount -af | |
| root@mfsbsd:~ # zfs set mountpoint=/ zroot/ROOT/default | |
| root@mfsbsd:~ # zfs set mountpoint=/tmp zroot/tmp | |
| root@mfsbsd:~ # zfs set mountpoint=/home zroot/home | |
| root@mfsbsd:~ # zfs set mountpoint=/usr zroot/usr | |
| root@mfsbsd:~ # zfs set mountpoint=/usr/local zroot/usr/local | |
| root@mfsbsd:~ # zfs set mountpoint=/usr/ports zroot/usr/ports | |
| root@mfsbsd:~ # zfs set mountpoint=/usr/ports/distfiles zroot/usr/ports/distfiles | |
| root@mfsbsd:~ # zfs set mountpoint=/usr/ports/packages zroot/usr/ports/packages | |
| root@mfsbsd:~ # zfs set mountpoint=/usr/src zroot/usr/src | |
| root@mfsbsd:~ # zfs set mountpoint=/var zroot/var | |
| root@mfsbsd:~ # zfs set mountpoint=/var/crash zroot/var/crash | |
| root@mfsbsd:~ # zfs set mountpoint=/var/db zroot/var/db | |
| root@mfsbsd:~ # zfs set mountpoint=/var/db/pkg zroot/var/db/pkg | |
| root@mfsbsd:~ # zfs set mountpoint=/var/empty zroot/var/empty | |
| root@mfsbsd:~ # zfs set mountpoint=/var/log zroot/var/log | |
| root@mfsbsd:~ # zfs set mountpoint=/var/mail zroot/var/mail | |
| root@mfsbsd:~ # zfs set mountpoint=/var/run zroot/var/run | |
| root@mfsbsd:~ # zfs set mountpoint=/var/tmp zroot/var/tmp | |
| root@mfsbsd:~ # zfs set canmount=noauto zroot/ROOT/default | |
| # Set booting from zboot/boot dataset. | |
| root@mfsbsd:~ # zpool set bootfs=zroot/ROOT/default zroot | |
| root@mfsbsd:~ # sync | |
| root@mfsbsd:~ # shutdown -r now |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
144-154 lines:
Using
fetch, you can download ready-made archives from ftp.http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/
or
http://ftp.freebsd.org/pub/FreeBSD/snapshots/amd64/13.1-STABLE/