Skip to content

Instantly share code, notes, and snippets.

@linux4life798

linux4life798/btrfs.md

Last active May 26, 2025 02:05
Show Gist options
  • Save linux4life798/72674b986b69cf66dfb125e715094f7d to your computer and use it in GitHub Desktop.
Save linux4life798/72674b986b69cf66dfb125e715094f7d to your computer and use it in GitHub Desktop.
Download ZIP
Resources for BTRFS Setup and Management
Raw
btrfs.md

BTRFS Notes

Packages

  • btrfs-progs
  • duperemove

Suggested BTRFS subvol as / layout for Snapper

https://wiki.archlinux.org/title/Snapper#Suggested_filesystem_layout

This allows for rolling the rootfs back to a snapshotted version, since the rootfs isn't tied to vol-id 5, anymore.

Debian installer uses /@rootfs as the subvol /, instead of @. See https://wiki.debian.org/Btrfs%20migration for more information.

When you need to make /@ level changes, you will still need to mount subvol-id 5 in /mnt.

sudo mount /dev/nvme0n1p3 /mnt
Ubuntu 24.04 ZFS install directory sub-volume layout

Ubuntu 24.04 has added installer support for ZFS. The directory/sub-volumene layout needed to support ZFS as a root filesystem is similar to that of BTRFS for roto filesystem.

$ sudo zfs list 
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
bpool                                             98.1M  1.65G    96K  /boot
bpool/BOOT                                        97.5M  1.65G    96K  none
bpool/BOOT/ubuntu_0gclb0                          97.4M  1.65G  97.4M  /boot
rpool                                             7.56G  1.73T   192K  /
rpool/ROOT                                        5.18G  1.73T   192K  none
rpool/ROOT/ubuntu_0gclb0                          5.18G  1.73T  3.69G  /
rpool/ROOT/ubuntu_0gclb0/srv                       192K  1.73T   192K  /srv
rpool/ROOT/ubuntu_0gclb0/usr                       576K  1.73T   192K  /usr
rpool/ROOT/ubuntu_0gclb0/usr/local                 384K  1.73T   384K  /usr/local
rpool/ROOT/ubuntu_0gclb0/var                      1.49G  1.73T   192K  /var
rpool/ROOT/ubuntu_0gclb0/var/games                 192K  1.73T   192K  /var/games
rpool/ROOT/ubuntu_0gclb0/var/lib                  1.48G  1.73T  1.33G  /var/lib
rpool/ROOT/ubuntu_0gclb0/var/lib/AccountsService   212K  1.73T   212K  /var/lib/AccountsService
rpool/ROOT/ubuntu_0gclb0/var/lib/NetworkManager    428K  1.73T   428K  /var/lib/NetworkManager
rpool/ROOT/ubuntu_0gclb0/var/lib/apt               100M  1.73T   100M  /var/lib/apt
rpool/ROOT/ubuntu_0gclb0/var/lib/dpkg             53.2M  1.73T  53.2M  /var/lib/dpkg
rpool/ROOT/ubuntu_0gclb0/var/log                  10.6M  1.73T  10.6M  /var/log
rpool/ROOT/ubuntu_0gclb0/var/mail                  192K  1.73T   192K  /var/mail
rpool/ROOT/ubuntu_0gclb0/var/snap                 2.65M  1.73T  2.65M  /var/snap
rpool/ROOT/ubuntu_0gclb0/var/spool                 276K  1.73T   276K  /var/spool
rpool/ROOT/ubuntu_0gclb0/var/www                   192K  1.73T   192K  /var/www
rpool/USERDATA                                    2.35G  1.73T   192K  none
rpool/USERDATA/home_z7g1tz                        2.34G  1.73T  2.34G  /home
rpool/USERDATA/root_z7g1tz                         532K  1.73T   532K  /root
rpool/keystore                                    22.5M  1.73T  16.4M  -
rpool/var                                          980K  1.73T   192K  /var
rpool/var/lib                                      788K  1.73T   192K  /var/lib
rpool/var/lib/docker                               596K  1.73T   596K  /var/lib/docker

Also see https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/SysadminGuide.html.

Resources

Debian Rootfs Install

When installing directly on a LUKs volume, you will need to use the installation shell to manually setup the install /target.

Manually initialize a GPT partiion table with the following:

Size Purpose GPT FS Type GPT Flags GPT Partition Name FS/Volume Name/Label
1 GiB EFI/ESP fat32 boot,esp efi EFI
2 GiB boot btrfs boot boot-btrfs
remainder LUKs luks Name inner FS root-btrfs.
Partitioning Commands

Install Utils

sudo apt install cryptsetup btrfs-progs dosfstools mtools nvme-cli smartmontools

Configure NVME SSD LBA

We want to select the most optimial (usually largest) logical block size for your SSD.

# Show offered LBA formats:
sudo nvme id-ns -H /dev/nvme0n1 | grep ^LBA
# Reformat/select a different format:
sudo nvme format /dev/nvme0n1 --lbaf=1
# You should probably reboot if you changed this block size.

See nvme-block-size-tuning.md for more info.

Cryptographically Erase the SSD

If your SSD supports OPAL for self encryption, you can optionally ask the SSD to erase itself. This would simply have the SSD regenerate it's encryption key for the whole drive.

sudo nvme format /dev/nvme0n1 --ses=2

See sed-nvme-cli-test.md for more information.

Partition

sudo parted -a optimal -s /dev/nvme0n1 -- unit MiB \
    mklabel gpt \
    mkpart efi fat32 1MiB 1025MiB \
    mkpart boot btrfs 1025MiB 3073MiB \
    mkpart luks 3073MiB '100%' \
    set 1 boot on \
    set 1 esp on \
    print unit s print
sudo parted -a optimal -s /dev/nvme0n1 -- unit MiB \
    align-check optimal 1 \
    align-check optimal 2 \
    align-check optimal 3
sudo partprobe -s

Format

The remaining steps can be done through the Debian installer.

sudo mkfs.fat -F 32 -n EFI /dev/nvme0n1p1    # Can force block size: -S 4096
sudo mkfs.btrfs -L boot-btrfs /dev/nvme0n1p2 # Can force block size: -s 4096
sudo cryptsetup luksFormat /dev/nvme0n1p3    # Can force block size: --sector-size 4096

sudo cryptsetup open /dev/nvme0n1p3 root-crypt
# Close with:
# sudo cryptsetup close root-crypt
sudo mkfs.btrfs -L root-btrfs /dev/mapper/root-crypt # Can force block size: -s 4096

Verify Partitions

# Show EFI FAT32 filesystem info:
sudo fsck.fat -v -n /dev/nvme0n1p1
# OR
# sudo minfo -i /dev/nvme0n1p1
sudo cryptsetup luksDump /dev/nvme0n1p3
# Additional block sizes can be seen in the output of blkid:
sudo blkid

Modifications

  • Label the main/root btrfs filesystem root-btrfs and the boot filesystem boot-btrfs.
Disable Copy-on-Write (CoW) for VM images 
sudo chattr +C  /var/lib/libvirt/images
sudo chattr +C ~/.local/share/libvirt/images
sudo chattr +C ~/.local/share/gnome-boxes/images
Setup SWAP space 
sudo mount /dev/disk/by-label/root-btrfs -osubvolid=5 /mnt
sudo btrfs subvolume create /mnt/@swap
sudo chmod u=rwx,g=,o= /mnt/@swap
sudo umount /mnt
sudo mkdir /var/swap
sudo chmod u=rwx,g=,o= /var/swap

# Edit /etc/fstab to mount @swap to /var/swap.
sudo systemctl daemon-reload
sudo mount /var/swap

sudo btrfs -v filesystem mkswapfile -s 8g /var/swap/swapfile1
sudo ls -al /var/swap

# Edit /etc/fstab and add the following line:
# /var/swap/swapfile1 none swap defaults 0 0
sudo swapon -a
sudo swapon

# Example mount line for the swap subvol:
# sudo mount /dev/disk/by-label/root-btrfs -osubvol=@swap /var/swap
Add Mirrored Drive 
sudo btrfs device add -f /dev/mapper/root2_crypt /
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /

Maintenance

Check health on storage server

sudo btrfs filesystem show
sudo btrfs device stats -c / || echo ERROR

Note -T is broken. See kdave/btrfs-progs#965.

Other maintenance actions

sudo btrfs scrub start /
sudo btrfs scrub status /
sudo btrfs balance start --background --full-balance /
sudo btrfs balance status /
sudo btrfs filesystem defragment -r /
Deduplication 
sudo duperemove -rdh /
# Or, you can speed things up with a hashfile, which will then
# use the save hash if the timestamp matches.
sudo duperemove -rdh -A --hashfile=/duperemove_hashfile /

# The ultimate version of this might look like the following:
sudo duperemove -rdh -A --hashfile=/duperemove_hashfile -b 4K --dedupe-options=partial --dedupe-options=same /
# --io-threads=$(($(nproc) * 8)) --cpu-threads=$(($(nproc) * 8))

# The duperemove tool can struggle to dedupe identical larger files (like identical 6GB ISOs),
# since it operates at the extents level. This can be helpful to get other gains, but not
# for perfect identical file deduplication.
# The fdupes command can identify them and then duperemove can combine them.
# Checkout the FAQ in https://markfasheh.github.io/duperemove/duperemove.html.
fdupes -c -r / | duperemove -hA --fdupes
# Might need --minsize=0
  • The -A allows duperemove to work over readonly snapshots.
  • The -c fdupes option enables the use of a cache/db to save hashes based on path+timestamp.

Warning: Running duperemove cannot effectively remege/resnapshot two unrelated subvolumes/snapshots. I did the following experiment, to test this. Restore root and home from a backup, manually using rsync. I then used btrfs send/receive to send the same backup as a subvolume to the machine being restored. I ran duperemove many times, but the most storage space I could recover/merge was about 200GB of 800GB. I still had a massive 600GB of undedupeable additional storage being consumed by this subvol that contains identical data to my main subvols.

Send and Receive Snapshots

# Send
sudo btrfs send -v --proto 0 /.snapshots/1/snapshot | dd status=progress | xz --threads=0 -c - | ssh HOST 'cat >/var/hdd/backups/btrfs-recv'
# Add "-p /path/to/prev/snapshot" to do incremental snapshot send.

Progress here will show uncompressed/raw data sent.

# Receive
mkfifo /var/hdd/backups/btrfs-recv
dd if=/var/hdd/backups/btrfs-recv status=progress | xz -d --threads=0 | sudo btrfs receive /var/hdd/backups/NAME/DATE/

Progress here will show the compressed data receive, so this will give you a feel for the efficiency of the compression. Using a fifo file allows us to transfer the data over ssh using our limited user, but run btrfs receive as root.

Interesting BTRFS Related Packages

@linux4life798
Copy link
Author

Note to self: I made a whole-file dedupe Go program that perfectly dedupes files from an non-parent snapshot. When you run btrfs filesystem du -s , it shows that all data is perfectly shared for all files.

It was based on the test example from https://go-review.googlesource.com/c/sys/+/284352.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment