Just some benchmarking using `fio` for my homelab.

Use fio to benchmark disk.

fio --randrepeat=1 --ioengine=libaio --direct=1 \
    --gtod_reduce=1 --name=test --bs=4k \
    --iodepth=64 --readwrite=randrw \
    --rwmixread=75 --size=1G \
    --filename=/mnt/path/to/test/target/deleteme

TL;DR Results

1st Place: NVME 2nd Place: NFS over Raid6 HDD 3rd Place: Raid

NVME SDD:    [r=314MiB/s, w=104MiB/s][r=80,500, w=26,600 IOPS]
NVME VirtIO: [r=199MiB/s, w= 66MiB/s][r=50,971, w=26,827 IOPS]
NFS RAID6:   [r= 88MiB/s, w= 29MiB/s][r=30,039, w= 7,509 IOPS]
RAID6 HDD:   [r=  7MiB/s, w=  2MiB/s][r= 1,765, w=   591 IOPS]

NVME Result Details

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=314MiB/s,w=104MiB/s][r=80.5k,w=26.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=462848: Fri Mar 11 08:58:06 2022
  read: IOPS=80.2k, BW=313MiB/s (328MB/s)(3070MiB/9802msec)
   bw (  KiB/s): min=302048, max=330080, per=100.00%, avg=321008.00, stdev=6728.88, samples=19
   iops        : min=75510, max=82520, avg=80252.00, stdev=1682.55, samples=19
  write: IOPS=26.8k, BW=105MiB/s (110MB/s)(1026MiB/9802msec); 0 zone resets
   bw (  KiB/s): min=102064, max=111976, per=100.00%, avg=107309.89, stdev=2369.17, samples=19
   iops        : min=25516, max=27994, avg=26827.47, stdev=592.29, samples=19
  cpu          : usr=26.12%, sys=53.22%, ctx=299871, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=3070MiB (3219MB), run=9802-9802msec
  WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=1026MiB (1076MB), run=9802-9802msec

Disk stats (read/write):
    dm-1: ios=779300/260607, merge=0/0, ticks=555264/2896, in_queue=558160, util=99.05%, aggrios=785926/262837, aggrmerge=0/135, aggrticks=558954/3686, aggrin_queue=562746, aggrutil=98.58%
  nvme0n1: ios=785926/262837, merge=0/135, ticks=558954/3686, in_queue=562746, util=98.58%

NVME VirtIO

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=202MiB/s,w=67.4MiB/s][r=51.8k,w=17.3k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=519953: Fri Mar 11 17:26:45 2022
  read: IOPS=51.0k, BW=199MiB/s (209MB/s)(768MiB/3850msec)
   bw (  KiB/s): min=189080, max=221896, per=99.87%, avg=203885.71, stdev=10204.06, samples=7
   iops        : min=47270, max=55474, avg=50971.43, stdev=2551.01, samples=7
  write: IOPS=17.1k, BW=66.6MiB/s (69.8MB/s)(256MiB/3850msec); 0 zone resets
   bw (  KiB/s): min=63296, max=73320, per=99.79%, avg=68062.86, stdev=3072.95, samples=7
   iops        : min=15824, max=18330, avg=17015.71, stdev=768.24, samples=7
  cpu          : usr=19.43%, sys=69.68%, ctx=40551, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=196498,65646,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=768MiB (805MB), run=3850-3850msec
  WRITE: bw=66.6MiB/s (69.8MB/s), 66.6MiB/s-66.6MiB/s (69.8MB/s-69.8MB/s), io=256MiB (269MB), run=3850-3850msec

Disk stats (read/write):
  sda: ios=189507/63349, merge=0/1, ticks=15743/5455, in_queue=21197, util=97.51%

NFS Raid6 HDD

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [f(1)][100.0%][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=372022: Fri Mar 11 09:59:07 2022
  read: IOPS=22.5k, BW=87.8MiB/s (92.1MB/s)(768MiB/8743msec)
   bw (  KiB/s): min=81112, max=94952, per=100.00%, avg=89919.53, stdev=3862.90, samples=17
   iops        : min=20278, max=23738, avg=22479.88, stdev=965.72, samples=17
  write: IOPS=7508, BW=29.3MiB/s (30.8MB/s)(256MiB/8743msec); 0 zone resets
   bw (  KiB/s): min=27512, max=31728, per=100.00%, avg=30039.53, stdev=1238.68, samples=17
   iops        : min= 6878, max= 7932, avg=7509.88, stdev=309.67, samples=17
  cpu          : usr=9.35%, sys=29.25%, ctx=87001, majf=0, minf=6
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=196498,65646,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=87.8MiB/s (92.1MB/s), 87.8MiB/s-87.8MiB/s (92.1MB/s-92.1MB/s), io=768MiB (805MB), run=8743-8743msec
  WRITE: bw=29.3MiB/s (30.8MB/s), 29.3MiB/s-29.3MiB/s (30.8MB/s-30.8MB/s), io=256MiB (269MB), run=8743-8743msec

Raid6 Spinning Disk Result Details

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=7063KiB/s,w=2366KiB/s][r=1765,w=591 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=426510: Fri Mar 11 07:24:01 2022
  read: IOPS=1387, BW=5551KiB/s (5685kB/s)(3070MiB/566284msec)
   bw (  KiB/s): min= 2677, max=11824, per=100.00%, avg=5555.57, stdev=879.39, samples=1132
   iops        : min=  669, max= 2956, avg=1388.83, stdev=219.85, samples=1132
  write: IOPS=463, BW=1855KiB/s (1900kB/s)(1026MiB/566284msec); 0 zone resets
   bw (  KiB/s): min=  889, max= 4208, per=100.00%, avg=1856.74, stdev=302.28, samples=1132
   iops        : min=  222, max= 1052, avg=464.10, stdev=75.61, samples=1132
  cpu          : usr=1.60%, sys=5.80%, ctx=933821, majf=0, minf=18
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=5551KiB/s (5685kB/s), 5551KiB/s-5551KiB/s (5685kB/s-5685kB/s), io=3070MiB (3219MB), run=566284-566284msec
  WRITE: bw=1855KiB/s (1900kB/s), 1855KiB/s-1855KiB/s (1900kB/s-1900kB/s), io=1026MiB (1076MB), run=566284-566284msec

Disk stats (read/write):
  sda: ios=785919/262701, merge=0/3, ticks=34968878/1219761, in_queue=36188640, util=100.00%

Conclusion

NVME is soooooooooo fast. Get it if you can. NFS makes read/writes faster. I don't know if it'll have the same effect layered over the NVME. RAID6 on HDD spinning disks is slooooooooooooooooooow. NVME VirtIO has 30% penalty with native NVME :(

References

https://docs.gitlab.com/ee/administration/operations/filesystem_benchmarking.html

ddrscott/nvme-ssd-vs-raid6.md