Tested volumes:
/dev/nvme1n1(type: st1, size: 125gb, defaults)/dev/nvme2n1(type: gp3, size: 15gb, iops:3000, tp: 125MB/s)/dev/nvme3n1(type: gp2, size: 15gb, defaults (iops:100 burstable until 3000))
All volumes created from scrach, no gp2->gp3 conversion.
EC2 instance: c6g.medium (ARM64)
Command:
fio --filename=device --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonlyiops-test-job: (groupid=0, jobs=4): err= 0: pid=4661: Tue Dec 8 10:49:36 2020
read: IOPS=3024, BW=11.8MiB/s (12.4MB/s)(1418MiB/120022msec)
slat (nsec): min=1583, max=28615k, avg=1320862.80, stdev=3295638.45
clat (msec): min=11, max=456, avg=337.16, stdev=32.67
lat (msec): min=15, max=456, avg=338.48, stdev=32.93
clat percentiles (msec):
| 1.00th=[ 239], 5.00th=[ 313], 10.00th=[ 317], 20.00th=[ 326],
| 30.00th=[ 330], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 347],
| 70.00th=[ 347], 80.00th=[ 355], 90.00th=[ 363], 95.00th=[ 368],
| 99.00th=[ 384], 99.50th=[ 388], 99.90th=[ 405], 99.95th=[ 414],
| 99.99th=[ 426]
bw ( KiB/s): min= 2512, max= 7656, per=24.93%, avg=3016.63, stdev=289.06, samples=960
iops : min= 628, max= 1914, avg=754.14, stdev=72.27, samples=960
lat (msec) : 20=0.02%, 50=0.66%, 100=0.09%, 250=0.26%, 500=98.98%
cpu : usr=0.08%, sys=0.21%, ctx=64783, majf=0, minf=1061
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=363064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=1418MiB (1487MB), run=120022-120022msec
Disk stats (read/write):
nvme2n1: ios=362690/0, merge=8/0, ticks=7101384/0, in_queue=3716744, util=99.97%
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4814: Tue Dec 8 10:53:21 2020
read: IOPS=3024, BW=11.8MiB/s (12.4MB/s)(1418MiB/120021msec)
slat (nsec): min=1551, max=10056k, avg=1321031.72, stdev=3245633.28
clat (msec): min=12, max=429, avg=337.15, stdev=28.93
lat (msec): min=21, max=429, avg=338.47, stdev=29.22
clat percentiles (msec):
| 1.00th=[ 234], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338],
| 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338],
| 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347],
| 99.00th=[ 363], 99.50th=[ 376], 99.90th=[ 384], 99.95th=[ 393],
| 99.99th=[ 401]
bw ( KiB/s): min= 2960, max= 7424, per=24.93%, avg=3016.29, stdev=265.77, samples=960
iops : min= 740, max= 1856, avg=754.04, stdev=66.45, samples=960
lat (msec) : 20=0.01%, 50=0.68%, 100=0.09%, 250=0.26%, 500=98.97%
cpu : usr=0.13%, sys=0.13%, ctx=52170, majf=0, minf=1064
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=363063,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=1418MiB (1487MB), run=120021-120021msec
Disk stats (read/write):
nvme3n1: ios=362693/0, merge=5/0, ticks=7067684/0, in_queue=3716776, util=99.96%
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4821: Tue Dec 8 10:56:57 2020
read: IOPS=33, BW=133KiB/s (136kB/s)(15.8MiB/121878msec)
slat (nsec): min=1673, max=862315k, avg=118776383.50, stdev=294236412.63
clat (msec): min=1090, max=33871, avg=27379.07, stdev=7467.12
lat (msec): min=1939, max=33940, avg=27497.85, stdev=7470.28
clat percentiles (msec):
| 1.00th=[ 3037], 5.00th=[ 7886], 10.00th=[14026], 20.00th=[17113],
| 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
| 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
| 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
| 99.99th=[17113]
bw ( KiB/s): min= 8, max= 56, per=41.64%, avg=55.38, stdev= 4.85, samples=438
iops : min= 2, max= 14, avg=13.81, stdev= 1.23, samples=438
cpu : usr=0.00%, sys=0.00%, ctx=620, majf=0, minf=1061
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=4055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=133KiB/s (136kB/s), 133KiB/s-133KiB/s (136kB/s-136kB/s), io=15.8MiB (16.6MB), run=121878-121878msec
Disk stats (read/write):
nvme1n1: ios=4047/0, merge=0/0, ticks=7122841/0, in_queue=3763204, util=99.98%
Command:
fio --filename=device --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonlyiops-test-job: (groupid=0, jobs=4): err= 0: pid=4829: Tue Dec 8 11:00:48 2020
read: IOPS=32.3k, BW=126MiB/s (132MB/s)(14.8GiB/120032msec)
slat (nsec): min=886, max=1143.3k, avg=1122.71, stdev=2157.57
clat (usec): min=55, max=61412, avg=31682.93, stdev=3278.13
lat (usec): min=208, max=61414, avg=31684.12, stdev=3278.02
clat percentiles (usec):
| 1.00th=[26870], 5.00th=[28967], 10.00th=[29754], 20.00th=[30540],
| 30.00th=[31065], 40.00th=[31589], 50.00th=[31851], 60.00th=[32113],
| 70.00th=[32637], 80.00th=[33424], 90.00th=[34341], 95.00th=[34866],
| 99.00th=[36439], 99.50th=[36963], 99.90th=[38011], 99.95th=[39060],
| 99.99th=[46400]
bw ( KiB/s): min=30760, max=100344, per=25.00%, avg=32313.87, stdev=4162.99, samples=960
iops : min= 7690, max=25086, avg=8078.44, stdev=1040.75, samples=960
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.31%, 4=0.46%, 10=0.13%, 20=0.01%, 50=99.08%
lat (msec) : 100=0.01%
cpu : usr=0.60%, sys=0.92%, ctx=133721, majf=0, minf=1066
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3879094,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=14.8GiB (15.9GB), run=120032-120032msec
Disk stats (read/write):
nvme2n1: ios=127820/0, merge=3747085/0, ticks=3998439/0, in_queue=3716864, util=99.98%
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4835: Tue Dec 8 11:03:33 2020
read: IOPS=33.1k, BW=129MiB/s (136MB/s)(15.1GiB/120031msec)
slat (nsec): min=894, max=932759, avg=1100.08, stdev=2211.40
clat (usec): min=1132, max=130793, avg=30951.44, stdev=3393.65
lat (usec): min=1143, max=130794, avg=30952.61, stdev=3393.52
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32],
| 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32],
| 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 32], 95.00th=[ 32],
| 99.00th=[ 32], 99.50th=[ 32], 99.90th=[ 32], 99.95th=[ 91],
| 99.99th=[ 123]
bw ( KiB/s): min=31776, max=103656, per=25.00%, avg=33078.53, stdev=4248.62, samples=960
iops : min= 7944, max=25914, avg=8269.61, stdev=1062.16, samples=960
lat (msec) : 2=0.02%, 4=0.99%, 10=0.02%, 20=0.02%, 50=98.88%
lat (msec) : 100=0.03%, 250=0.03%
cpu : usr=0.58%, sys=0.96%, ctx=127979, majf=0, minf=1069
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3970844,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=15.1GiB (16.3GB), run=120031-120031msec
Disk stats (read/write):
nvme3n1: ios=128104/0, merge=3839219/0, ticks=3955260/0, in_queue=3716512, util=99.98%
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4842: Tue Dec 8 11:06:10 2020
read: IOPS=4667, BW=18.2MiB/s (19.1MB/s)(2191MiB/120155msec)
slat (nsec): min=878, max=1029.5k, avg=1156.79, stdev=2561.85
clat (usec): min=632, max=727669, avg=219374.82, stdev=116999.26
lat (usec): min=634, max=727670, avg=219376.05, stdev=116999.24
clat percentiles (msec):
| 1.00th=[ 108], 5.00th=[ 115], 10.00th=[ 118], 20.00th=[ 122],
| 30.00th=[ 126], 40.00th=[ 146], 50.00th=[ 178], 60.00th=[ 213],
| 70.00th=[ 266], 80.00th=[ 326], 90.00th=[ 397], 95.00th=[ 456],
| 99.00th=[ 550], 99.50th=[ 575], 99.90th=[ 651], 99.95th=[ 667],
| 99.99th=[ 726]
bw ( KiB/s): min= 824, max= 8176, per=24.99%, avg=4664.47, stdev=1583.94, samples=960
iops : min= 206, max= 2044, avg=1166.09, stdev=395.99, samples=960
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.06%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.34%, 250=68.27%, 500=28.47%, 750=2.82%
cpu : usr=0.08%, sys=0.15%, ctx=20417, majf=0, minf=1071
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=560810,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=2191MiB (2297MB), run=120155-120155msec
Disk stats (read/write):
nvme1n1: ios=18631/0, merge=541660/0, ticks=4024160/0, in_queue=3721936, util=99.98%
Command:
fio --filename=device --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 iops-test-job: (groupid=0, jobs=4): err= 0: pid=4973: Tue Dec 8 12:05:32 2020
read: IOPS=1512, BW=6051KiB/s (6196kB/s)(709MiB/120023msec)
slat (nsec): min=1493, max=38620k, avg=1332134.10, stdev=3313125.49
clat (msec): min=13, max=391, avg=332.73, stdev=28.43
lat (msec): min=21, max=391, avg=334.06, stdev=28.70
clat percentiles (msec):
| 1.00th=[ 236], 5.00th=[ 321], 10.00th=[ 326], 20.00th=[ 326],
| 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 338], 60.00th=[ 338],
| 70.00th=[ 338], 80.00th=[ 342], 90.00th=[ 347], 95.00th=[ 347],
| 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 372], 99.95th=[ 372],
| 99.99th=[ 380]
bw ( KiB/s): min= 1224, max= 3560, per=24.94%, avg=1508.72, stdev=152.24, samples=960
iops : min= 306, max= 890, avg=377.17, stdev=38.06, samples=960
write: IOPS=1512, BW=6049KiB/s (6194kB/s)(709MiB/120023msec); 0 zone resets
slat (nsec): min=1551, max=37140k, avg=1309680.21, stdev=3287985.96
clat (msec): min=12, max=424, avg=341.56, stdev=31.59
lat (msec): min=21, max=433, avg=342.87, stdev=31.85
clat percentiles (msec):
| 1.00th=[ 213], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 334],
| 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 347], 60.00th=[ 347],
| 70.00th=[ 351], 80.00th=[ 355], 90.00th=[ 363], 95.00th=[ 368],
| 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 397], 99.95th=[ 401],
| 99.99th=[ 414]
bw ( KiB/s): min= 1264, max= 3752, per=24.93%, avg=1508.03, stdev=159.78, samples=960
iops : min= 316, max= 938, avg=377.00, stdev=39.95, samples=960
lat (msec) : 20=0.01%, 50=0.70%, 100=0.09%, 250=0.26%, 500=98.95%
cpu : usr=0.10%, sys=0.19%, ctx=56486, majf=0, minf=43
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=181557,181511,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=6051KiB/s (6196kB/s), 6051KiB/s-6051KiB/s (6196kB/s-6196kB/s), io=709MiB (744MB), run=120023-120023msec
WRITE: bw=6049KiB/s (6194kB/s), 6049KiB/s-6049KiB/s (6194kB/s-6194kB/s), io=709MiB (743MB), run=120023-120023msec
Disk stats (read/write):
nvme2n1: ios=181429/181305, merge=0/4, ticks=2703582/4375786, in_queue=3716896, util=100.00%
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4981: Tue Dec 8 12:07:59 2020
read: IOPS=1512, BW=6051KiB/s (6196kB/s)(709MiB/120023msec)
slat (nsec): min=1461, max=24547k, avg=1326464.65, stdev=3252327.87
clat (msec): min=17, max=382, avg=332.96, stdev=27.30
lat (msec): min=21, max=382, avg=334.28, stdev=27.58
clat percentiles (msec):
| 1.00th=[ 224], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 338],
| 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338],
| 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 347],
| 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355],
| 99.99th=[ 363]
bw ( KiB/s): min= 1248, max= 3648, per=24.94%, avg=1508.61, stdev=150.10, samples=960
iops : min= 312, max= 912, avg=377.15, stdev=37.53, samples=960
write: IOPS=1512, BW=6049KiB/s (6194kB/s)(709MiB/120023msec); 0 zone resets
slat (nsec): min=1501, max=25373k, avg=1315415.85, stdev=3239952.20
clat (msec): min=12, max=438, avg=341.32, stdev=30.52
lat (msec): min=21, max=438, avg=342.63, stdev=30.78
clat percentiles (msec):
| 1.00th=[ 201], 5.00th=[ 326], 10.00th=[ 338], 20.00th=[ 338],
| 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 347], 60.00th=[ 347],
| 70.00th=[ 347], 80.00th=[ 355], 90.00th=[ 363], 95.00th=[ 363],
| 99.00th=[ 376], 99.50th=[ 384], 99.90th=[ 393], 99.95th=[ 393],
| 99.99th=[ 405]
bw ( KiB/s): min= 1280, max= 3776, per=24.93%, avg=1508.07, stdev=157.66, samples=960
iops : min= 320, max= 944, avg=377.01, stdev=39.42, samples=960
lat (msec) : 20=0.01%, 50=0.71%, 100=0.09%, 250=0.26%, 500=98.93%
cpu : usr=0.09%, sys=0.19%, ctx=52286, majf=0, minf=41
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=181555,181512,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=6051KiB/s (6196kB/s), 6051KiB/s-6051KiB/s (6196kB/s-6196kB/s), io=709MiB (744MB), run=120023-120023msec
WRITE: bw=6049KiB/s (6194kB/s), 6049KiB/s-6049KiB/s (6194kB/s-6194kB/s), io=709MiB (743MB), run=120023-120023msec
Disk stats (read/write):
nvme3n1: ios=181420/181317, merge=0/2, ticks=2687327/4382966, in_queue=3716784, util=99.98%
iops-test-job: (groupid=0, jobs=4): err= 0: pid=4989: Tue Dec 8 12:10:31 2020
read: IOPS=15, BW=63.0KiB/s (65.5kB/s)(7800KiB/121908msec)
slat (nsec): min=1583, max=1484.3M, avg=118236233.82, stdev=294966037.99
clat (msec): min=1088, max=33091, avg=27159.57, stdev=7317.49
lat (msec): min=1939, max=33091, avg=27277.81, stdev=7323.22
clat percentiles (msec):
| 1.00th=[ 2802], 5.00th=[ 7752], 10.00th=[14563], 20.00th=[17113],
| 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
| 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
| 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
| 99.99th=[17113]
bw ( KiB/s): min= 8, max= 56, per=44.64%, avg=28.12, stdev=10.94, samples=418
iops : min= 2, max= 14, avg= 7.00, stdev= 2.74, samples=418
write: IOPS=16, BW=67.8KiB/s (69.4kB/s)(8264KiB/121908msec); 0 zone resets
slat (usec): min=2, max=860490, avg=121581.24, stdev=296881.20
clat (msec): min=1545, max=35727, avg=27575.66, stdev=7685.45
lat (msec): min=1939, max=35727, avg=27697.24, stdev=7685.57
clat percentiles (msec):
| 1.00th=[ 2970], 5.00th=[ 7886], 10.00th=[13489], 20.00th=[17113],
| 30.00th=[17113], 40.00th=[17113], 50.00th=[17113], 60.00th=[17113],
| 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113],
| 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
| 99.99th=[17113]
bw ( KiB/s): min= 8, max= 56, per=42.75%, avg=28.64, stdev=11.52, samples=425
iops : min= 2, max= 14, avg= 7.12, stdev= 2.90, samples=425
cpu : usr=0.00%, sys=0.00%, ctx=626, majf=0, minf=41
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=1950,2066,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=63.0KiB/s (65.5kB/s), 63.0KiB/s-63.0KiB/s (65.5kB/s-65.5kB/s), io=7800KiB (7987kB), run=121908-121908msec
WRITE: bw=67.8KiB/s (69.4kB/s), 67.8KiB/s-67.8KiB/s (69.4kB/s-69.4kB/s), io=8264KiB (8462kB), run=121908-121908msec
Disk stats (read/write):
nvme1n1: ios=2001/2065, merge=0/0, ticks=2678905/4457067, in_queue=3765564, util=100.00%
Command:
fio --filename=device --direct=1 --rw=randread --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonlythroughput-test-job: (groupid=0, jobs=4): err= 0: pid=4848: Tue Dec 8 11:11:19 2020
read: IOPS=2019, BW=126MiB/s (132MB/s)(14.8GiB/120032msec)
slat (nsec): min=1698, max=39413k, avg=1978836.02, stdev=4887117.00
clat (usec): min=1538, max=238269, avg=124756.76, stdev=15294.44
lat (usec): min=1540, max=248438, avg=126735.70, stdev=16157.69
clat percentiles (msec):
| 1.00th=[ 88], 5.00th=[ 111], 10.00th=[ 112], 20.00th=[ 113],
| 30.00th=[ 125], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 127],
| 70.00th=[ 127], 80.00th=[ 129], 90.00th=[ 140], 95.00th=[ 142],
| 99.00th=[ 167], 99.50th=[ 169], 99.90th=[ 197], 99.95th=[ 199],
| 99.99th=[ 218]
bw ( KiB/s): min=29952, max=97280, per=24.98%, avg=32290.26, stdev=3782.80, samples=960
iops : min= 468, max= 1520, avg=504.53, stdev=59.11, samples=960
lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.80%, 50=0.08%
lat (msec) : 100=0.29%, 250=98.80%
cpu : usr=0.05%, sys=0.17%, ctx=37717, majf=0, minf=2085
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=242441,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=14.8GiB (15.9GB), run=120032-120032msec
Disk stats (read/write):
nvme2n1: ios=242102/0, merge=67/0, ticks=7077244/0, in_queue=3716776, util=99.96%
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4868: Tue Dec 8 11:15:42 2020
read: IOPS=2067, BW=129MiB/s (135MB/s)(15.1GiB/120031msec)
slat (nsec): min=1773, max=56015k, avg=1933148.44, stdev=4753362.85
clat (msec): min=6, max=232, avg=121.88, stdev=14.60
lat (msec): min=6, max=245, avg=123.81, stdev=15.45
clat percentiles (msec):
| 1.00th=[ 93], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110],
| 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124],
| 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 138], 95.00th=[ 138],
| 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 192], 99.95th=[ 192],
| 99.99th=[ 220]
bw ( KiB/s): min=32191, max=97920, per=24.98%, avg=33050.74, stdev=3866.76, samples=960
iops : min= 502, max= 1530, avg=516.36, stdev=60.42, samples=960
lat (msec) : 10=0.02%, 20=0.84%, 50=0.05%, 100=0.11%, 250=98.98%
cpu : usr=0.05%, sys=0.17%, ctx=35666, majf=0, minf=2093
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=248169,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=15.1GiB (16.3GB), run=120031-120031msec
Disk stats (read/write):
nvme3n1: ios=247840/0, merge=58/0, ticks=7068356/0, in_queue=3716776, util=99.97%
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4874: Tue Dec 8 11:18:05 2020
read: IOPS=33, BW=2129KiB/s (2180kB/s)(253MiB/121857msec)
slat (nsec): min=1854, max=862349k, avg=118777048.51, stdev=294241889.92
clat (msec): min=1119, max=10195, avg=7438.29, stdev=1105.93
lat (msec): min=1708, max=11017, avg=7557.07, stdev=1144.82
clat percentiles (msec):
| 1.00th=[ 2567], 5.00th=[ 5738], 10.00th=[ 6812], 20.00th=[ 6812],
| 30.00th=[ 6812], 40.00th=[ 7617], 50.00th=[ 7617], 60.00th=[ 7617],
| 70.00th=[ 7617], 80.00th=[ 8490], 90.00th=[ 8490], 95.00th=[ 8490],
| 99.00th=[ 9329], 99.50th=[ 9329], 99.90th=[10134], 99.95th=[10134],
| 99.99th=[10134]
bw ( KiB/s): min= 128, max= 896, per=41.79%, avg=889.62, stdev=66.58, samples=547
iops : min= 2, max= 14, avg=13.87, stdev= 1.06, samples=547
cpu : usr=0.00%, sys=0.00%, ctx=635, majf=0, minf=2089
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4054,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=2129KiB/s (2180kB/s), 2129KiB/s-2129KiB/s (2180kB/s-2180kB/s), io=253MiB (266MB), run=121857-121857msec
Disk stats (read/write):
nvme1n1: ios=4047/0, merge=0/0, ticks=7122982/0, in_queue=3762548, util=99.97%
Command:
fio --filename=device --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4926: Tue Dec 8 11:52:46 2020
read: IOPS=1006, BW=62.9MiB/s (65.0MB/s)(7555MiB/120046msec)
slat (nsec): min=1821, max=39453k, avg=1986510.31, stdev=4889550.93
clat (msec): min=3, max=616, avg=118.01, stdev=12.81
lat (msec): min=3, max=616, avg=120.00, stdev=13.73
clat percentiles (msec):
| 1.00th=[ 86], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 112],
| 30.00th=[ 112], 40.00th=[ 113], 50.00th=[ 114], 60.00th=[ 126],
| 70.00th=[ 126], 80.00th=[ 127], 90.00th=[ 127], 95.00th=[ 128],
| 99.00th=[ 140], 99.50th=[ 142], 99.90th=[ 155], 99.95th=[ 155],
| 99.99th=[ 169]
bw ( KiB/s): min=12544, max=46336, per=24.98%, avg=16099.06, stdev=2081.48, samples=960
iops : min= 196, max= 724, avg=251.53, stdev=32.52, samples=960
write: IOPS=1012, BW=63.3MiB/s (66.4MB/s)(7598MiB/120046msec); 0 zone resets
slat (usec): min=2, max=39904, avg=1971.03, stdev=4875.74
clat (msec): min=2, max=237, avg=131.47, stdev=20.49
lat (msec): min=2, max=249, avg=133.44, stdev=21.13
clat percentiles (msec):
| 1.00th=[ 80], 5.00th=[ 111], 10.00th=[ 112], 20.00th=[ 114],
| 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 127], 60.00th=[ 130],
| 70.00th=[ 140], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 167],
| 99.00th=[ 182], 99.50th=[ 186], 99.90th=[ 199], 99.95th=[ 209],
| 99.99th=[ 224]
bw ( KiB/s): min=12672, max=52352, per=24.98%, avg=16190.79, stdev=2205.70, samples=960
iops : min= 198, max= 818, avg=252.96, stdev=34.46, samples=960
lat (msec) : 4=0.01%, 10=0.01%, 20=0.63%, 50=0.28%, 100=0.61%
lat (msec) : 250=98.46%, 750=0.01%
cpu : usr=0.09%, sys=0.17%, ctx=37373, majf=0, minf=45
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=120876,121574,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=62.9MiB/s (65.0MB/s), 62.9MiB/s-62.9MiB/s (65.0MB/s-65.0MB/s), io=7555MiB (7922MB), run=120046-120046msec
WRITE: bw=63.3MiB/s (66.4MB/s), 63.3MiB/s-63.3MiB/s (66.4MB/s-66.4MB/s), io=7598MiB (7967MB), run=120046-120046msec
Disk stats (read/write):
nvme2n1: ios=120775/121406, merge=9/21, ticks=2697966/4378140, in_queue=3716808, util=99.99
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4933: Tue Dec 8 11:55:14 2020
read: IOPS=1031, BW=64.4MiB/s (67.6MB/s)(7735MiB/120038msec)
slat (nsec): min=1838, max=37381k, avg=1908799.09, stdev=4725123.56
clat (msec): min=9, max=164, avg=115.24, stdev=11.85
lat (msec): min=9, max=177, avg=117.15, stdev=12.75
clat percentiles (msec):
| 1.00th=[ 49], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 110],
| 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 111], 60.00th=[ 123],
| 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 124], 95.00th=[ 124],
| 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 150], 99.95th=[ 150],
| 99.99th=[ 163]
bw ( KiB/s): min=12416, max=46464, per=24.98%, avg=16482.66, stdev=2140.85, samples=960
iops : min= 194, max= 726, avg=257.52, stdev=33.45, samples=960
write: IOPS=1036, BW=64.8MiB/s (67.9MB/s)(7776MiB/120038msec); 0 zone resets
slat (usec): min=2, max=33424, avg=1957.05, stdev=4774.57
clat (msec): min=12, max=232, avg=128.47, stdev=19.62
lat (msec): min=12, max=232, avg=130.42, stdev=20.26
clat percentiles (msec):
| 1.00th=[ 39], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 111],
| 30.00th=[ 123], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 125],
| 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 150], 95.00th=[ 165],
| 99.00th=[ 178], 99.50th=[ 180], 99.90th=[ 192], 99.95th=[ 205],
| 99.99th=[ 220]
bw ( KiB/s): min=13568, max=49920, per=24.98%, avg=16571.29, stdev=2250.00, samples=960
iops : min= 212, max= 780, avg=258.91, stdev=35.16, samples=960
lat (msec) : 10=0.01%, 20=0.03%, 50=1.00%, 100=0.25%, 250=98.72%
cpu : usr=0.11%, sys=0.16%, ctx=36183, majf=0, minf=43
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=123765,124418,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=64.4MiB/s (67.6MB/s), 64.4MiB/s-64.4MiB/s (67.6MB/s-67.6MB/s), io=7735MiB (8111MB), run=120038-120038msec
WRITE: bw=64.8MiB/s (67.9MB/s), 64.8MiB/s-64.8MiB/s (67.9MB/s-67.9MB/s), io=7776MiB (8154MB), run=120038-120038msec
Disk stats (read/write):
nvme3n1: ios=123660/124258, merge=8/15, ticks=2661447/4411715, in_queue=3716812, util=99.98%
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=4941: Tue Dec 8 11:57:54 2020
read: IOPS=16, BW=1025KiB/s (1050kB/s)(122MiB/121995msec)
slat (nsec): min=1862, max=1490.1M, avg=118153243.49, stdev=294777046.18
clat (msec): min=1459, max=9424, avg=7031.26, stdev=896.27
lat (msec): min=1970, max=9515, avg=7149.41, stdev=933.58
clat percentiles (msec):
| 1.00th=[ 2769], 5.00th=[ 5940], 10.00th=[ 6745], 20.00th=[ 6812],
| 30.00th=[ 6812], 40.00th=[ 6812], 50.00th=[ 6812], 60.00th=[ 7617],
| 70.00th=[ 7617], 80.00th=[ 7617], 90.00th=[ 7617], 95.00th=[ 7617],
| 99.00th=[ 8490], 99.50th=[ 8557], 99.90th=[ 9329], 99.95th=[ 9463],
| 99.99th=[ 9463]
bw ( KiB/s): min= 127, max= 1024, per=43.93%, avg=450.27, stdev=199.23, samples=523
iops : min= 1, max= 16, avg= 6.99, stdev= 3.13, samples=523
write: IOPS=16, BW=1086KiB/s (1112kB/s)(129MiB/121995msec); 0 zone resets
slat (usec): min=3, max=2693.6k, avg=121362.88, stdev=301987.09
clat (msec): min=1120, max=12701, avg=7861.34, stdev=1320.27
lat (msec): min=1970, max=12701, avg=7982.70, stdev=1342.85
clat percentiles (msec):
| 1.00th=[ 2836], 5.00th=[ 6342], 10.00th=[ 6812], 20.00th=[ 6812],
| 30.00th=[ 7617], 40.00th=[ 7617], 50.00th=[ 7617], 60.00th=[ 7617],
| 70.00th=[ 8490], 80.00th=[ 8490], 90.00th=[ 9329], 95.00th=[10134],
| 99.00th=[10939], 99.50th=[11073], 99.90th=[11879], 99.95th=[11879],
| 99.99th=[12684]
bw ( KiB/s): min= 127, max= 896, per=44.08%, avg=478.25, stdev=190.90, samples=517
iops : min= 1, max= 14, avg= 7.43, stdev= 2.98, samples=517
cpu : usr=0.01%, sys=0.00%, ctx=631, majf=0, minf=41
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.2%, >=64=93.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1954,2070,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1025KiB/s (1050kB/s), 1025KiB/s-1025KiB/s (1050kB/s-1050kB/s), io=122MiB (128MB), run=121995-121995msec
WRITE: bw=1086KiB/s (1112kB/s), 1086KiB/s-1086KiB/s (1112kB/s-1112kB/s), io=129MiB (136MB), run=121995-121995msec
Disk stats (read/write):
nvme1n1: ios=2003/2067, merge=0/0, ticks=2621725/4511318, in_queue=3767852, util=100.00%
fio --filename=device --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=1 --numjobs=1 --time_based --group_reporting --name=readlatency-test-job --runtime=120 --eta-newline=1readlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5002: Tue Dec 8 12:14:58 2020
read: IOPS=1511, BW=6046KiB/s (6191kB/s)(708MiB/120002msec)
slat (nsec): min=2191, max=26814, avg=2417.91, stdev=412.79
clat (usec): min=137, max=6898, avg=658.42, stdev=880.73
lat (usec): min=139, max=6901, avg=660.94, stdev=880.71
clat percentiles (usec):
| 1.00th=[ 161], 5.00th=[ 167], 10.00th=[ 172], 20.00th=[ 182],
| 30.00th=[ 198], 40.00th=[ 281], 50.00th=[ 322], 60.00th=[ 343],
| 70.00th=[ 392], 80.00th=[ 898], 90.00th=[ 1827], 95.00th=[ 2769],
| 99.00th=[ 4424], 99.50th=[ 4752], 99.90th=[ 4948], 99.95th=[ 4948],
| 99.99th=[ 5014]
bw ( KiB/s): min= 5184, max= 6968, per=100.00%, avg=6045.35, stdev=321.84, samples=240
iops : min= 1296, max= 1742, avg=1511.33, stdev=80.46, samples=240
lat (usec) : 250=37.55%, 500=36.35%, 750=3.72%, 1000=4.03%
lat (msec) : 2=9.48%, 4=7.21%, 10=1.66%
cpu : usr=0.33%, sys=0.45%, ctx=181453, majf=0, minf=9
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=181372,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=6046KiB/s (6191kB/s), 6046KiB/s-6046KiB/s (6191kB/s-6191kB/s), io=708MiB (743MB), run=120002-120002msec
Disk stats (read/write):
nvme2n1: ios=181227/0, merge=0/0, ticks=119280/0, in_queue=119032, util=99.23%
readlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5008: Tue Dec 8 12:21:18 2020
read: IOPS=3024, BW=11.8MiB/s (12.4MB/s)(1418MiB/120001msec)
slat (nsec): min=2215, max=156932, avg=2401.67, stdev=435.21
clat (nsec): min=1584, max=19853k, avg=327461.06, stdev=105418.30
lat (usec): min=132, max=19855, avg=329.95, stdev=105.41
clat percentiles (usec):
| 1.00th=[ 157], 5.00th=[ 176], 10.00th=[ 194], 20.00th=[ 231],
| 30.00th=[ 289], 40.00th=[ 314], 50.00th=[ 326], 60.00th=[ 338],
| 70.00th=[ 359], 80.00th=[ 420], 90.00th=[ 461], 95.00th=[ 482],
| 99.00th=[ 529], 99.50th=[ 570], 99.90th=[ 783], 99.95th=[ 873],
| 99.99th=[ 1221]
bw ( KiB/s): min=11976, max=16144, per=100.00%, avg=12099.72, stdev=603.68, samples=239
iops : min= 2994, max= 4036, avg=3024.93, stdev=150.92, samples=239
lat (usec) : 2=0.01%, 250=23.44%, 500=74.16%, 750=2.27%, 1000=0.10%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=0.66%, sys=0.88%, ctx=363151, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=362996,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=1418MiB (1487MB), run=120001-120001msec
Disk stats (read/write):
nvme3n1: ios=362696/0, merge=0/0, ticks=118713/0, in_queue=119476, util=99.60%
readlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5099: Tue Dec 8 14:25:22 2020
read: IOPS=33, BW=133KiB/s (136kB/s)(15.6MiB/120003msec)
slat (nsec): min=2314, max=26839, avg=4111.66, stdev=1294.69
clat (usec): min=97, max=70677, avg=30047.33, stdev=5787.25
lat (usec): min=100, max=70682, avg=30051.64, stdev=5787.20
clat percentiles (usec):
| 1.00th=[ 8848], 5.00th=[18744], 10.00th=[25560], 20.00th=[29754],
| 30.00th=[30278], 40.00th=[30278], 50.00th=[30278], 60.00th=[30278],
| 70.00th=[30278], 80.00th=[30802], 90.00th=[34866], 95.00th=[40109],
| 99.00th=[46400], 99.50th=[49021], 99.90th=[54264], 99.95th=[63701],
| 99.99th=[70779]
bw ( KiB/s): min= 127, max= 384, per=100.00%, avg=133.03, stdev=16.77, samples=240
iops : min= 31, max= 96, avg=33.23, stdev= 4.20, samples=240
lat (usec) : 100=0.03%, 250=0.35%, 500=0.05%, 750=0.08%, 1000=0.03%
lat (msec) : 2=0.05%, 4=0.15%, 10=0.33%, 20=4.61%, 50=94.01%
lat (msec) : 100=0.33%
cpu : usr=0.01%, sys=0.01%, ctx=3995, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3993,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=133KiB/s (136kB/s), 133KiB/s-133KiB/s (136kB/s-136kB/s), io=15.6MiB (16.4MB), run=120003-120003msec
Disk stats (read/write):
nvme1n1: ios=3989/0, merge=0/0, ticks=119854/0, in_queue=119852, util=99.91%
fio --filename=device --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=1 --numjobs=1 --time_based --group_reporting --name=readlatency-test-job --runtime=120 --eta-newline=1rwlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5102: Tue Dec 8 14:28:21 2020
read: IOPS=680, BW=2721KiB/s (2787kB/s)(319MiB/120001msec)
slat (nsec): min=2190, max=27544, avg=2436.86, stdev=433.44
clat (usec): min=145, max=5660, avg=652.73, stdev=878.30
lat (usec): min=148, max=5663, avg=655.26, stdev=878.28
clat percentiles (usec):
| 1.00th=[ 161], 5.00th=[ 167], 10.00th=[ 169], 20.00th=[ 178],
| 30.00th=[ 188], 40.00th=[ 251], 50.00th=[ 322], 60.00th=[ 338],
| 70.00th=[ 396], 80.00th=[ 881], 90.00th=[ 1811], 95.00th=[ 2737],
| 99.00th=[ 4424], 99.50th=[ 4752], 99.90th=[ 4948], 99.95th=[ 4948],
| 99.99th=[ 5080]
bw ( KiB/s): min= 2184, max= 3168, per=100.00%, avg=2721.06, stdev=154.96, samples=240
iops : min= 546, max= 792, avg=680.24, stdev=38.77, samples=240
write: IOPS=677, BW=2712KiB/s (2777kB/s)(318MiB/120001msec); 0 zone resets
slat (nsec): min=2265, max=169813, avg=2656.61, stdev=740.21
clat (usec): min=298, max=31654, avg=813.37, stdev=850.47
lat (usec): min=322, max=31657, avg=816.12, stdev=850.44
clat percentiles (usec):
| 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 408],
| 30.00th=[ 424], 40.00th=[ 453], 50.00th=[ 474], 60.00th=[ 506],
| 70.00th=[ 586], 80.00th=[ 906], 90.00th=[ 1844], 95.00th=[ 2802],
| 99.00th=[ 4490], 99.50th=[ 4817], 99.90th=[ 5014], 99.95th=[ 6980],
| 99.99th=[11207]
bw ( KiB/s): min= 2288, max= 3112, per=100.00%, avg=2711.29, stdev=146.42, samples=240
iops : min= 572, max= 778, avg=677.80, stdev=36.62, samples=240
lat (usec) : 250=20.02%, 500=46.40%, 750=10.97%, 1000=4.20%
lat (msec) : 2=9.53%, 4=7.19%, 10=1.69%, 20=0.01%, 50=0.01%
cpu : usr=0.35%, sys=0.38%, ctx=163062, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=81638,81346,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2721KiB/s (2787kB/s), 2721KiB/s-2721KiB/s (2787kB/s-2787kB/s), io=319MiB (334MB), run=120001-120001msec
WRITE: bw=2712KiB/s (2777kB/s), 2712KiB/s-2712KiB/s (2777kB/s-2777kB/s), io=318MiB (333MB), run=120001-120001msec
Disk stats (read/write):
nvme2n1: ios=81628/81260, merge=0/0, ticks=53288/66077, in_queue=119200, util=99.36%
rwlatency-test-job: (groupid=0, jobs=1): err= 0: pid=5107: Tue Dec 8 14:31:30 2020
read: IOPS=1400, BW=5600KiB/s (5735kB/s)(656MiB/120001msec)
slat (nsec): min=2240, max=183344, avg=2429.48, stdev=570.94
clat (nsec): min=1386, max=28928k, avg=231410.77, stdev=114596.33
lat (usec): min=127, max=28930, avg=233.94, stdev=114.60
clat percentiles (usec):
| 1.00th=[ 145], 5.00th=[ 153], 10.00th=[ 157], 20.00th=[ 163],
| 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 192], 60.00th=[ 265],
| 70.00th=[ 281], 80.00th=[ 302], 90.00th=[ 326], 95.00th=[ 347],
| 99.00th=[ 441], 99.50th=[ 502], 99.90th=[ 742], 99.95th=[ 865],
| 99.99th=[ 1909]
bw ( KiB/s): min= 4624, max= 6440, per=100.00%, avg=5600.59, stdev=310.01, samples=239
iops : min= 1156, max= 1610, avg=1400.15, stdev=77.50, samples=239
write: IOPS=1399, BW=5597KiB/s (5731kB/s)(656MiB/120001msec); 0 zone resets
slat (nsec): min=2289, max=35142, avg=2573.56, stdev=373.34
clat (usec): min=339, max=25415, avg=476.60, stdev=116.37
lat (usec): min=341, max=25418, avg=479.27, stdev=116.38
clat percentiles (usec):
| 1.00th=[ 383], 5.00th=[ 400], 10.00th=[ 408], 20.00th=[ 420],
| 30.00th=[ 433], 40.00th=[ 445], 50.00th=[ 457], 60.00th=[ 474],
| 70.00th=[ 490], 80.00th=[ 519], 90.00th=[ 570], 95.00th=[ 619],
| 99.00th=[ 725], 99.50th=[ 783], 99.90th=[ 963], 99.95th=[ 1172],
| 99.99th=[ 3851]
bw ( KiB/s): min= 4736, max= 6208, per=100.00%, avg=5596.25, stdev=241.27, samples=239
iops : min= 1184, max= 1552, avg=1399.06, stdev=60.32, samples=239
lat (usec) : 2=0.01%, 250=28.40%, 500=58.11%, 750=13.07%, 1000=0.36%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=0.57%, sys=0.89%, ctx=336069, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=168012,167907,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=5600KiB/s (5735kB/s), 5600KiB/s-5600KiB/s (5735kB/s-5735kB/s), io=656MiB (688MB), run=120001-120001msec
WRITE: bw=5597KiB/s (5731kB/s), 5597KiB/s-5597KiB/s (5731kB/s-5731kB/s), io=656MiB (688MB), run=120001-120001msec
Disk stats (read/write):
nvme3n1: ios=167916/167761, merge=0/0, ticks=38846/79935, in_queue=118460, util=98.75%