I/O Benchmarking and NVME/Encryption/FS Tuning

0-io-block-size-tuning-and-benchmarking.md

Here is the cheatsheet for block size tuning of NVME drives, LUKS encryption, filesystem, and partitions.

dd-io-benchmarking.md

The GNOME Disks App - https://wiki.gnome.org/Design/Apps/Disks

It uses 100.0 MiB "Sample size". I am interpreting this as the blocks size. The key is the direct flag.

Read Performance:

sudo dd if=/dev/nvme0n1p3 of=/dev/null bs=100M status=progress iflag=direct

ext-block-size.md

sudo tune2fs -l /dev/nvme0n1p2 | grep "^Block size"
# Block size:               4096

luks-sector-side.md

The cryptsetup utility will set the luks volume's sector side to whatever the disk reports. This is usually 512 bytes by default, but would be 4K if you were able to reconfigure your nvme to use 4K LBA.

If you can't set your NVME to 4K LBA, but still want to use 4K block size with LUKS, you can override the "sector size" by doing the following:

sudo cryptsetup luksFormat --sector-size 4096 /dev/nvme0n1p3
# The --label option is another good to note one.

# Check:
sudo cryptsetup luksDump /dev/nvme0n1p3

There are some additional cryptsetup tuning options, like same-cpu-crypt and submit-from-crypt-cpus.

systemd/systemd#11946

nvme-block-size-tuning.md

Check what block/LBA sizes are supported by your nvme:

sudo smartctl -a /dev/nvme0n1
# ...
# Supported LBA Sizes (NSID 0x1)
# Id Fmt  Data  Metadt  Rel_Perf
#  0 +     512       0         2
#  1 -    4096       0         1
# ...

# AND/OR

sudo nvme id-ns -H /dev/nvme0n1 | grep ^LBA
# LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
# LBA Format  1 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better

Change to possibly larger 4K LBA:

sudo nvme format /dev/nvme0 --lbaf=1

# Check reported block size:
lsblk -t

THIS WILL ERASE ALL DATA

---

See https://wiki.archlinux.org/title/Advanced_Format#NVMe_solid_state_drives.

partition-alignment.md

From https://wiki.archlinux.org/title/Advanced_Format:

Aligning partitions correctly avoids excessive read-modify-write cycles. A typical practice for personal computers is to have each partition's start and size aligned to 1 MiB (1 048 576 bytes) marks. This covers all common page and block size scenarios, as it is divisible by all commonly used sizes—1 MiB, 512 KiB, 128 KiB, 4 KiB, and 512 B.

TL;DR Align start and size¹ to 1MiB and 4MiB for cheap flash drives ².

sudo fdisk -l /dev/nvme0n1

# Disk /dev/nvme0n1: ...
# Disk model: ...               
# Units: sectors of 1 * 512 = 512 bytes
# Sector size (logical/physical): 512 bytes / 512 bytes
# I/O size (minimum/optimal): 512 bytes / 512 bytes
# Disklabel type: gpt
# Disk identifier: ...
#
# Device           Start        End    Sectors   Size Type
# /dev/nvme0n1p1    2048    2099199    2097152     1G EFI System
# /dev/nvme0n1p2 2099200    6293503    4194304     2G Linux filesystem
# /dev/nvme0n1p3 6293504 2000408575 1994115072 950.9G Linux filesystem

This NVME still has it's LBA size set to 512.

• Checking the first partition's start offset:

  $$512 \text{ }{}^\text{bytes}/{}_\text{block} \times 2048 \text{ blocks} \times \frac{1 \text{ MiB}}{1024 \times 1024 \text{ bytes}} = 1 \text{MiB}$$

  This is definitley a multiple of 1MiB.

• Checking the third partition's start offset:

  $$512 \text{ }{}^\text{bytes}/{}_\text{block} \times 6293504 \text{ blocks} = 3222274048 \text{ bytes} = 3073 \text{ MiB}$$

  This cleanly became a multiple of MiB, without decimal/fraction.

Footnotes

1. The Arch wiki for parted, https://wiki.archlinux.org/title/Parted#Alignment, infers that the size needs to be aligned, aswell, for luks/dm-crypt to function correctly. Also checkout https://wiki.archlinux.org/title/Advanced_Format#Partition_alignment, which nicely suggests just defining start and end ranges in MiB (command unit MiB switches units) to ensure easy 1MiB alignment. ↩

2. Parted manual says that 1MiB alignement might not be anough for cheap flash drives and you should use 4MiB, instead. See https://www.gnu.org/software/parted/manual/parted.html#FOOT2 and https://lwn.net/Articles/428584/. ↩