Setup:
# create a ramdisk, and an encryption key
mount -t tmpfs -o size=1024M none /mem
fallocate -l 1024M /mem/file
echo myzfskey > /tmp/zfskey
Create a pool:
# none
zpool create -O atime=off -O compression=off -O encryption=off mypool /mem/file
# aesgcm
zpool create -O atime=off -O compression=off -O encryption=aes-256-gcm -O keylocation=file:///tmp/zfskey -O keyformat=passphrase mypool /mem/file
# chapoly
zpool create -O atime=off -O compression=off -O encryption=chacha20-poly1305 -O keylocation=file:///tmp/zfskey -O keyformat=passphrase mypool /mem/file
# zstd
zpool create -O atime=off -O compression=zstd -O encryption=off mypool /mem/file
# zaesgcm
zpool create -O atime=off -O compression=zstd -O encryption=aes-256-gcm -O keylocation=file:///tmp/zfskey -O keyformat=passphrase mypool /mem/file
# zchapoly
zpool create -O atime=off -O compression=zstd -O encryption=chacha20-poly1305 -O keylocation=file:///tmp/zfskey -O keyformat=passphrase mypool /mem/file
Test it:
fio --name=xxx --size=4K --bs=4K --rw=randrw --ioengine=psync --sync=1 --iodepth=32 --numjobs=1 --direct=1 --end_fsync=1 --gtod_reduce=1 --time_based --runtime=60
none: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
none: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=5721KiB/s,w=5549KiB/s][r=1430,w=1387 IOPS][eta 00m:00s]
none: (groupid=0, jobs=1): err= 0: pid=1484: Sun Nov 27 16:39:52 2022
read: IOPS=2729, BW=10.7MiB/s (11.2MB/s)(640MiB/60020msec)
bw ( KiB/s): min= 4696, max=36672, per=100.00%, avg=10967.06, stdev=8013.82, samples=119
iops : min= 1174, max= 9168, avg=2741.76, stdev=2003.45, samples=119
write: IOPS=2727, BW=10.7MiB/s (11.2MB/s)(640MiB/60020msec); 0 zone resets
bw ( KiB/s): min= 5088, max=36440, per=100.00%, avg=10961.41, stdev=7902.80, samples=119
iops : min= 1272, max= 9110, avg=2740.35, stdev=1975.70, samples=119
cpu : usr=6.14%, sys=40.37%, ctx=327507, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=163816,163734,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=10.7MiB/s (11.2MB/s), 10.7MiB/s-10.7MiB/s (11.2MB/s-11.2MB/s), io=640MiB (671MB), run=60020-60020msec
WRITE: bw=10.7MiB/s (11.2MB/s), 10.7MiB/s-10.7MiB/s (11.2MB/s-11.2MB/s), io=640MiB (671MB), run=60020-60020msec
aesgcm: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
aesgcm: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=3308KiB/s,w=3428KiB/s][r=827,w=857 IOPS][eta 00m:00s] ]
aesgcm: (groupid=0, jobs=1): err= 0: pid=2084: Sun Nov 27 16:40:55 2022
read: IOPS=945, BW=3782KiB/s (3873kB/s)(222MiB/60022msec)
bw ( KiB/s): min= 2544, max= 8624, per=100.00%, avg=3792.87, stdev=914.94, samples=119
iops : min= 636, max= 2156, avg=948.22, stdev=228.73, samples=119
write: IOPS=941, BW=3767KiB/s (3857kB/s)(221MiB/60022msec); 0 zone resets
bw ( KiB/s): min= 2896, max= 8344, per=100.00%, avg=3773.58, stdev=866.46, samples=119
iops : min= 724, max= 2086, avg=943.39, stdev=216.62, samples=119
cpu : usr=3.31%, sys=23.36%, ctx=113129, majf=0, minf=9
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=56754,56524,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=3782KiB/s (3873kB/s), 3782KiB/s-3782KiB/s (3873kB/s-3873kB/s), io=222MiB (232MB), run=60022-60022msec
WRITE: bw=3767KiB/s (3857kB/s), 3767KiB/s-3767KiB/s (3857kB/s-3857kB/s), io=221MiB (232MB), run=60022-60022msec
chapoly: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
chapoly: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=4292KiB/s,w=4080KiB/s][r=1073,w=1020 IOPS][eta 00m:00s]
chapoly: (groupid=0, jobs=1): err= 0: pid=150774: Fri Dec 2 11:16:11 2022
read: IOPS=1331, BW=5325KiB/s (5452kB/s)(312MiB/60024msec)
bw ( KiB/s): min= 3496, max=11864, per=100.00%, avg=5340.37, stdev=1423.99, samples=119
iops : min= 874, max= 2966, avg=1335.09, stdev=356.00, samples=119
write: IOPS=1324, BW=5299KiB/s (5427kB/s)(311MiB/60024msec); 0 zone resets
bw ( KiB/s): min= 4016, max=12024, per=100.00%, avg=5314.42, stdev=1423.07, samples=119
iops : min= 1004, max= 3006, avg=1328.61, stdev=355.77, samples=119
cpu : usr=4.55%, sys=30.94%, ctx=159119, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=79900,79522,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=5325KiB/s (5452kB/s), 5325KiB/s-5325KiB/s (5452kB/s-5452kB/s), io=312MiB (327MB), run=60024-60024msec
WRITE: bw=5299KiB/s (5427kB/s), 5299KiB/s-5299KiB/s (5427kB/s-5427kB/s), io=311MiB (326MB), run=60024-60024msec
znone: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
znone: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=12.9MiB/s,w=12.7MiB/s][r=3306,w=3251 IOPS][eta 00m:00s]
znone: (groupid=0, jobs=1): err= 0: pid=3303: Sun Nov 27 16:43:00 2022
read: IOPS=3465, BW=13.5MiB/s (14.2MB/s)(812MiB/60008msec)
bw ( KiB/s): min= 4336, max=36376, per=100.00%, avg=13915.50, stdev=9845.28, samples=119
iops : min= 1084, max= 9094, avg=3478.87, stdev=2461.32, samples=119
write: IOPS=3463, BW=13.5MiB/s (14.2MB/s)(812MiB/60008msec); 0 zone resets
bw ( KiB/s): min= 4984, max=36448, per=100.00%, avg=13909.58, stdev=9809.28, samples=119
iops : min= 1246, max= 9112, avg=3477.39, stdev=2452.32, samples=119
cpu : usr=6.73%, sys=41.60%, ctx=415709, majf=0, minf=6
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=207963,207843,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=13.5MiB/s (14.2MB/s), 13.5MiB/s-13.5MiB/s (14.2MB/s-14.2MB/s), io=812MiB (852MB), run=60008-60008msec
WRITE: bw=13.5MiB/s (14.2MB/s), 13.5MiB/s-13.5MiB/s (14.2MB/s-14.2MB/s), io=812MiB (851MB), run=60008-60008msec
zaesgcm: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
zaesgcm: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=3295KiB/s,w=3239KiB/s][r=823,w=809 IOPS][eta 00m:00s]
zaesgcm: (groupid=0, jobs=1): err= 0: pid=3978: Sun Nov 27 16:44:03 2022
read: IOPS=1104, BW=4419KiB/s (4525kB/s)(259MiB/60032msec)
bw ( KiB/s): min= 2712, max=12384, per=100.00%, avg=4431.80, stdev=2007.67, samples=119
iops : min= 678, max= 3096, avg=1107.95, stdev=501.92, samples=119
write: IOPS=1098, BW=4396KiB/s (4501kB/s)(258MiB/60032msec); 0 zone resets
bw ( KiB/s): min= 2864, max=11728, per=100.00%, avg=4408.34, stdev=1928.31, samples=119
iops : min= 716, max= 2932, avg=1102.08, stdev=482.08, samples=119
cpu : usr=3.30%, sys=23.59%, ctx=132000, majf=0, minf=5
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=66316,65970,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=4419KiB/s (4525kB/s), 4419KiB/s-4419KiB/s (4525kB/s-4525kB/s), io=259MiB (272MB), run=60032-60032msec
WRITE: bw=4396KiB/s (4501kB/s), 4396KiB/s-4396KiB/s (4501kB/s-4501kB/s), io=258MiB (270MB), run=60032-60032msec
zchapoly: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
zchapoly: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=4028KiB/s,w=4096KiB/s][r=1007,w=1024 IOPS][eta 00m:00s]
zchapoly: (groupid=0, jobs=1): err= 0: pid=152078: Fri Dec 2 11:23:46 2022
read: IOPS=1317, BW=5271KiB/s (5397kB/s)(309MiB/60010msec)
bw ( KiB/s): min= 3288, max=19296, per=100.00%, avg=5283.23, stdev=2298.62, samples=119
iops : min= 822, max= 4824, avg=1320.81, stdev=574.65, samples=119
write: IOPS=1311, BW=5245KiB/s (5371kB/s)(307MiB/60010msec); 0 zone resets
bw ( KiB/s): min= 3544, max=19544, per=100.00%, avg=5258.69, stdev=2243.66, samples=119
iops : min= 886, max= 4886, avg=1314.66, stdev=560.91, samples=119
cpu : usr=4.51%, sys=30.64%, ctx=157447, majf=0, minf=6
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=79073,78685,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=5271KiB/s (5397kB/s), 5271KiB/s-5271KiB/s (5397kB/s-5397kB/s), io=309MiB (324MB), run=60010-60010msec
WRITE: bw=5245KiB/s (5371kB/s), 5245KiB/s-5245KiB/s (5371kB/s-5371kB/s), io=307MiB (322MB), run=60010-60010msec
none: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
none: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=111MiB/s,w=112MiB/s][r=28.4k,w=28.7k IOPS][eta 00m:00s]
none: (groupid=0, jobs=1): err= 0: pid=3382405: Sun Nov 27 16:52:23 2022
read: IOPS=34.0k, BW=137MiB/s (143MB/s)(8202MiB/60001msec)
bw ( KiB/s): min=99936, max=163216, per=100.00%, avg=140241.21, stdev=16548.47, samples=119
iops : min=24984, max=40804, avg=35060.32, stdev=4137.11, samples=119
write: IOPS=34.0k, BW=137MiB/s (143MB/s)(8193MiB/60001msec); 0 zone resets
bw ( KiB/s): min=101824, max=162360, per=100.00%, avg=140071.19, stdev=16607.64, samples=119
iops : min=25456, max=40590, avg=35017.80, stdev=4151.91, samples=119
cpu : usr=6.01%, sys=52.46%, ctx=4194867, majf=0, minf=5
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2099649,2097405,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=8202MiB (8600MB), run=60001-60001msec
WRITE: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=8193MiB (8591MB), run=60001-60001msec
aesgcm: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
aesgcm: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=93.2MiB/s,w=93.0MiB/s][r=23.9k,w=24.1k IOPS][eta 00m:00s]
aesgcm: (groupid=0, jobs=1): err= 0: pid=3384339: Sun Nov 27 16:53:24 2022
read: IOPS=23.7k, BW=92.5MiB/s (97.0MB/s)(5551MiB/60002msec)
bw ( KiB/s): min=82800, max=103024, per=100.00%, avg=94749.18, stdev=3653.78, samples=119
iops : min=20700, max=25756, avg=23687.31, stdev=913.45, samples=119
write: IOPS=23.7k, BW=92.4MiB/s (96.9MB/s)(5544MiB/60002msec); 0 zone resets
bw ( KiB/s): min=83240, max=103032, per=100.00%, avg=94630.59, stdev=3535.18, samples=119
iops : min=20810, max=25758, avg=23657.65, stdev=883.79, samples=119
cpu : usr=4.93%, sys=45.48%, ctx=2838692, majf=0, minf=6
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1421042,1419340,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=92.5MiB/s (97.0MB/s), 92.5MiB/s-92.5MiB/s (97.0MB/s-97.0MB/s), io=5551MiB (5821MB), run=60002-60002msec
WRITE: bw=92.4MiB/s (96.9MB/s), 92.4MiB/s-92.4MiB/s (96.9MB/s-96.9MB/s), io=5544MiB (5814MB), run=60002-60002msec
chapoly: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=71.9MiB/s,w=71.2MiB/s][r=18.4k,w=18.2k IOPS][eta 00m:00s]
chapoly: (groupid=0, jobs=1): err= 0: pid=1860933: Fri Dec 2 19:37:16 2022
read: IOPS=19.9k, BW=77.7MiB/s (81.5MB/s)(4663MiB/60001msec)
bw ( KiB/s): min=62616, max=92056, per=100.00%, avg=79637.16, stdev=7560.42, samples=119
iops : min=15654, max=23014, avg=19909.29, stdev=1890.11, samples=119
write: IOPS=19.9k, BW=77.7MiB/s (81.4MB/s)(4660MiB/60001msec); 0 zone resets
bw ( KiB/s): min=64048, max=90952, per=100.00%, avg=79590.97, stdev=7497.55, samples=119
iops : min=16012, max=22738, avg=19897.74, stdev=1874.39, samples=119
cpu : usr=3.53%, sys=30.41%, ctx=2385830, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1193647,1192847,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=77.7MiB/s (81.5MB/s), 77.7MiB/s-77.7MiB/s (81.5MB/s-81.5MB/s), io=4663MiB (4889MB), run=60001-60001msec
WRITE: bw=77.7MiB/s (81.4MB/s), 77.7MiB/s-77.7MiB/s (81.4MB/s-81.4MB/s), io=4660MiB (4886MB), run=60001-60001msec
znone: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
znone: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=109MiB/s,w=108MiB/s][r=27.0k,w=27.7k IOPS][eta 00m:00s]
znone: (groupid=0, jobs=1): err= 0: pid=3387074: Sun Nov 27 16:55:27 2022
read: IOPS=28.8k, BW=112MiB/s (118MB/s)(6739MiB/60002msec)
bw ( KiB/s): min=99888, max=131232, per=100.00%, avg=115090.82, stdev=5099.94, samples=119
iops : min=24972, max=32808, avg=28772.72, stdev=1275.00, samples=119
write: IOPS=28.7k, BW=112MiB/s (118MB/s)(6732MiB/60002msec); 0 zone resets
bw ( KiB/s): min=103024, max=131208, per=100.00%, avg=115001.88, stdev=4901.21, samples=119
iops : min=25756, max=32802, avg=28750.49, stdev=1225.31, samples=119
cpu : usr=6.00%, sys=53.05%, ctx=3446969, majf=0, minf=5
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1725166,1723472,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=112MiB/s (118MB/s), 112MiB/s-112MiB/s (118MB/s-118MB/s), io=6739MiB (7066MB), run=60002-60002msec
WRITE: bw=112MiB/s (118MB/s), 112MiB/s-112MiB/s (118MB/s-118MB/s), io=6732MiB (7059MB), run=60002-60002msec
zaesgcm: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
zaesgcm: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=91.1MiB/s,w=91.0MiB/s][r=23.3k,w=23.5k IOPS][eta 00m:00s]
zaesgcm: (groupid=0, jobs=1): err= 0: pid=3388843: Sun Nov 27 16:56:28 2022
read: IOPS=23.7k, BW=92.4MiB/s (96.9MB/s)(5545MiB/60002msec)
bw ( KiB/s): min=70584, max=106144, per=100.00%, avg=94650.22, stdev=4780.09, samples=119
iops : min=17646, max=26536, avg=23662.55, stdev=1195.02, samples=119
write: IOPS=23.6k, BW=92.3MiB/s (96.8MB/s)(5538MiB/60002msec); 0 zone resets
bw ( KiB/s): min=71016, max=106256, per=100.00%, avg=94528.34, stdev=4623.31, samples=119
iops : min=17754, max=26564, avg=23632.08, stdev=1155.83, samples=119
cpu : usr=4.72%, sys=45.62%, ctx=2835712, majf=0, minf=6
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1419497,1417813,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=92.4MiB/s (96.9MB/s), 92.4MiB/s-92.4MiB/s (96.9MB/s-96.9MB/s), io=5545MiB (5814MB), run=60002-60002msec
WRITE: bw=92.3MiB/s (96.8MB/s), 92.3MiB/s-92.3MiB/s (96.8MB/s-96.8MB/s), io=5538MiB (5807MB), run=60002-60002msec
zchapoly: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
zchapoly: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=63.6MiB/s,w=64.6MiB/s][r=16.3k,w=16.5k IOPS][eta 00m:00s]
zchapoly: (groupid=0, jobs=1): err= 0: pid=1864007: Fri Dec 2 19:51:48 2022
read: IOPS=18.3k, BW=71.5MiB/s (74.0MB/s)(4291MiB/60002msec)
bw ( KiB/s): min=38000, max=89176, per=100.00%, avg=73286.18, stdev=10532.45, samples=119
iops : min= 9500, max=22294, avg=18321.55, stdev=2633.11, samples=119
write: IOPS=18.3k, BW=71.5MiB/s (74.0MB/s)(4289MiB/60002msec); 0 zone resets
bw ( KiB/s): min=38240, max=88192, per=100.00%, avg=73250.15, stdev=10342.04, samples=119
iops : min= 9560, max=22048, avg=18312.55, stdev=2585.50, samples=119
cpu : usr=3.67%, sys=30.62%, ctx=2196406, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1098412,1098066,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=71.5MiB/s (74.0MB/s), 71.5MiB/s-71.5MiB/s (74.0MB/s-74.0MB/s), io=4291MiB (4499MB), run=60002-60002msec
WRITE: bw=71.5MiB/s (74.0MB/s), 71.5MiB/s-71.5MiB/s (74.0MB/s-74.0MB/s), io=4289MiB (4498MB), run=60002-60002msec
none: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
none: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=22.8MiB/s,w=22.7MiB/s][r=5839,w=5822 IOPS][eta 00m:00s]
none: (groupid=0, jobs=1): err= 0: pid=3231: Sun Nov 27 05:37:26 2022
read: IOPS=5643, BW=22.0MiB/s (23.1MB/s)(1323MiB/60004msec)
bw ( KiB/s): min=18600, max=24752, per=100.00%, avg=22576.34, stdev=1306.60, samples=119
iops : min= 4650, max= 6188, avg=5644.08, stdev=326.65, samples=119
write: IOPS=5635, BW=22.0MiB/s (23.1MB/s)(1321MiB/60004msec); 0 zone resets
bw ( KiB/s): min=19104, max=23416, per=100.00%, avg=22547.23, stdev=1102.03, samples=119
iops : min= 4776, max= 5854, avg=5636.81, stdev=275.51, samples=119
cpu : usr=5.26%, sys=53.05%, ctx=684135, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=338605,338129,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=1323MiB (1387MB), run=60004-60004msec
WRITE: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=1321MiB (1385MB), run=60004-60004msec
aesgcm: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
aesgcm: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=6376KiB/s,w=6520KiB/s][r=1594,w=1630 IOPS][eta 00m:00s]
aesgcm: (groupid=0, jobs=1): err= 0: pid=4003: Sun Nov 27 05:38:30 2022
read: IOPS=1619, BW=6476KiB/s (6631kB/s)(380MiB/60016msec)
bw ( KiB/s): min= 5592, max= 7336, per=99.98%, avg=6475.56, stdev=338.16, samples=119
iops : min= 1398, max= 1834, avg=1618.89, stdev=84.54, samples=119
write: IOPS=1615, BW=6461KiB/s (6616kB/s)(379MiB/60016msec); 0 zone resets
bw ( KiB/s): min= 6144, max= 6544, per=100.00%, avg=6464.07, stdev=93.92, samples=119
iops : min= 1536, max= 1636, avg=1616.02, stdev=23.48, samples=119
cpu : usr=1.73%, sys=15.75%, ctx=194353, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=97167,96943,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=6476KiB/s (6631kB/s), 6476KiB/s-6476KiB/s (6631kB/s-6631kB/s), io=380MiB (398MB), run=60016-60016msec
WRITE: bw=6461KiB/s (6616kB/s), 6461KiB/s-6461KiB/s (6616kB/s-6616kB/s), io=379MiB (397MB), run=60016-60016msec
chapoly: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
chapoly: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=15.5MiB/s,w=15.5MiB/s][r=3970,w=3979 IOPS][eta 00m:00s]
chapoly: (groupid=0, jobs=1): err= 0: pid=150957: Fri Dec 2 01:32:35 2022
read: IOPS=3908, BW=15.3MiB/s (16.0MB/s)(916MiB/60004msec)
bw ( KiB/s): min=11648, max=17392, per=100.00%, avg=15642.89, stdev=983.65, samples=119
iops : min= 2912, max= 4348, avg=3910.72, stdev=245.91, samples=119
write: IOPS=3909, BW=15.3MiB/s (16.0MB/s)(916MiB/60004msec); 0 zone resets
bw ( KiB/s): min=12072, max=16120, per=100.00%, avg=15644.24, stdev=839.18, samples=119
iops : min= 3018, max= 4030, avg=3911.06, stdev=209.80, samples=119
cpu : usr=3.93%, sys=38.45%, ctx=470627, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=234555,234563,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=15.3MiB/s (16.0MB/s), 15.3MiB/s-15.3MiB/s (16.0MB/s-16.0MB/s), io=916MiB (961MB), run=60004-60004msec
WRITE: bw=15.3MiB/s (16.0MB/s), 15.3MiB/s-15.3MiB/s (16.0MB/s-16.0MB/s), io=916MiB (961MB), run=60004-60004msec
znone: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
znone: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=22.6MiB/s,w=22.7MiB/s][r=5787,w=5799 IOPS][eta 00m:00s]
znone: (groupid=0, jobs=1): err= 0: pid=5220: Sun Nov 27 05:40:38 2022
read: IOPS=5648, BW=22.1MiB/s (23.1MB/s)(1324MiB/60003msec)
bw ( KiB/s): min=18120, max=24816, per=100.00%, avg=22598.45, stdev=1299.65, samples=119
iops : min= 4530, max= 6204, avg=5649.61, stdev=324.91, samples=119
write: IOPS=5640, BW=22.0MiB/s (23.1MB/s)(1322MiB/60003msec); 0 zone resets
bw ( KiB/s): min=18576, max=23488, per=100.00%, avg=22571.43, stdev=1117.07, samples=119
iops : min= 4644, max= 5872, avg=5642.86, stdev=279.27, samples=119
cpu : usr=5.26%, sys=53.41%, ctx=680735, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=338932,338442,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=22.1MiB/s (23.1MB/s), 22.1MiB/s-22.1MiB/s (23.1MB/s-23.1MB/s), io=1324MiB (1388MB), run=60003-60003msec
WRITE: bw=22.0MiB/s (23.1MB/s), 22.0MiB/s-22.0MiB/s (23.1MB/s-23.1MB/s), io=1322MiB (1386MB), run=60003-60003msec
zaesgcm: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
zaesgcm: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=6662KiB/s,w=6526KiB/s][r=1665,w=1631 IOPS][eta 00m:00s]
zaesgcm: (groupid=0, jobs=1): err= 0: pid=6091: Sun Nov 27 05:41:42 2022
read: IOPS=1622, BW=6491KiB/s (6647kB/s)(380MiB/60009msec)
bw ( KiB/s): min= 5592, max= 7648, per=99.99%, avg=6491.50, stdev=349.75, samples=119
iops : min= 1398, max= 1912, avg=1622.87, stdev=87.44, samples=119
write: IOPS=1619, BW=6477KiB/s (6633kB/s)(380MiB/60009msec); 0 zone resets
bw ( KiB/s): min= 6024, max= 6568, per=100.00%, avg=6480.00, stdev=108.32, samples=119
iops : min= 1506, max= 1642, avg=1620.00, stdev=27.08, samples=119
cpu : usr=1.52%, sys=15.77%, ctx=194769, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=97386,97172,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=6491KiB/s (6647kB/s), 6491KiB/s-6491KiB/s (6647kB/s-6647kB/s), io=380MiB (399MB), run=60009-60009msec
WRITE: bw=6477KiB/s (6633kB/s), 6477KiB/s-6477KiB/s (6633kB/s-6633kB/s), io=380MiB (398MB), run=60009-60009msec
zchapoly: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
fio-3.25
Starting 1 process
zchapoly: Laying out IO file (1 file / 0MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=15.4MiB/s,w=15.6MiB/s][r=3943,w=4002 IOPS][eta 00m:00s]
zchapoly: (groupid=0, jobs=1): err= 0: pid=151867: Fri Dec 2 02:15:50 2022
read: IOPS=3785, BW=14.8MiB/s (15.5MB/s)(887MiB/60005msec)
bw ( KiB/s): min=10160, max=17528, per=100.00%, avg=15142.55, stdev=1637.63, samples=119
iops : min= 2540, max= 4382, avg=3785.64, stdev=409.41, samples=119
write: IOPS=3786, BW=14.8MiB/s (15.5MB/s)(887MiB/60005msec); 0 zone resets
bw ( KiB/s): min=10552, max=16160, per=100.00%, avg=15146.98, stdev=1604.75, samples=119
iops : min= 2638, max= 4040, avg=3786.74, stdev=401.19, samples=119
cpu : usr=3.74%, sys=38.71%, ctx=456350, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=227138,227196,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=14.8MiB/s (15.5MB/s), 14.8MiB/s-14.8MiB/s (15.5MB/s-15.5MB/s), io=887MiB (930MB), run=60005-60005msec
WRITE: bw=14.8MiB/s (15.5MB/s), 14.8MiB/s-14.8MiB/s (15.5MB/s-15.5MB/s), io=887MiB (931MB), run=60005-60005msec
That is really cool