fio --name=write_iops --size=10G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1
fio --name=write_throughput --numjobs=8 \
--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write \
--group_reporting=1
fio --name=read_iops --size=10G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1
fio --name=read_throughput --numjobs=8 \
--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read \
--group_reporting=1
| SEC 1TB | SN850 1TB | |
|---|---|---|
| 4k-random-write | 249.5 MiB/s | 262 MiB/s |
| 1M-sequential-write | 722.5 MiB/s | 979 MiB/s |
| 4K-random-read | 218.5 MiB/s | 235 MiB/s |
| 1M-sequential-read | 988 MiB/s | 1023 MiB/s |
Run 1:
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
write_iops: Laying out IO file (1 file / 10240MiB)
write_iops: (groupid=0, jobs=1): err= 0: pid=555361: Sat Mar 5 16:18:26 2022
write: IOPS=69.9k, BW=273MiB/s (286MB/s)(15.0GiB/60001msec); 0 zone resets
slat (nsec): min=1093, max=52085k, avg=13671.65, stdev=71591.99
clat (usec): min=254, max=56921, avg=901.96, stdev=645.34
lat (usec): min=258, max=56927, avg=915.69, stdev=650.79
clat percentiles (usec):
| 1.00th=[ 510], 5.00th=[ 578], 10.00th=[ 603], 20.00th=[ 660],
| 30.00th=[ 725], 40.00th=[ 799], 50.00th=[ 848], 60.00th=[ 930],
| 70.00th=[ 979], 80.00th=[ 1074], 90.00th=[ 1172], 95.00th=[ 1237],
| 99.00th=[ 2008], 99.50th=[ 2343], 99.90th=[ 7111], 99.95th=[ 8979],
| 99.99th=[36439]
bw ( KiB/s): min=244440, max=295872, per=100.00%, avg=279493.56, stdev=11282.74, samples=119
iops : min=61110, max=73968, avg=69873.34, stdev=2820.69, samples=119
lat (usec) : 500=0.63%, 750=34.02%, 1000=37.65%
lat (msec) : 2=26.71%, 4=0.88%, 10=0.08%, 20=0.01%, 50=0.02%
lat (msec) : 100=0.01%
cpu : usr=4.98%, sys=38.49%, ctx=397079, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,4192225,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=15.0GiB (17.2GB), run=60001-60001msec
Disk stats (read/write):
sda: ios=0/4313120, merge=0/55330, ticks=0/3417481, in_queue=3417622, util=99.86%
Run 2:
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
write_iops: (groupid=0, jobs=1): err= 0: pid=555879: Sat Mar 5 16:19:40 2022
write: IOPS=57.6k, BW=225MiB/s (236MB/s)(13.2GiB/60005msec); 0 zone resets
slat (nsec): min=1025, max=31872k, avg=16601.21, stdev=121301.79
clat (usec): min=239, max=34416, avg=1094.77, stdev=1003.48
lat (usec): min=244, max=34417, avg=1111.45, stdev=1012.11
clat percentiles (usec):
| 1.00th=[ 445], 5.00th=[ 586], 10.00th=[ 603], 20.00th=[ 709],
| 30.00th=[ 734], 40.00th=[ 840], 50.00th=[ 889], 60.00th=[ 971],
| 70.00th=[ 1074], 80.00th=[ 1188], 90.00th=[ 1582], 95.00th=[ 2114],
| 99.00th=[ 6325], 99.50th=[ 7963], 99.90th=[10683], 99.95th=[11469],
| 99.99th=[31327]
bw ( KiB/s): min=107640, max=288936, per=99.99%, avg=230229.71, stdev=57019.06, samples=120
iops : min=26910, max=72234, avg=57557.40, stdev=14254.76, samples=120
lat (usec) : 250=0.01%, 500=2.03%, 750=30.27%, 1000=31.57%
lat (msec) : 2=30.40%, 4=4.12%, 10=1.43%, 20=0.16%, 50=0.02%
cpu : usr=4.19%, sys=18.20%, ctx=402356, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,3454191,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=13.2GiB (14.1GB), run=60005-60005msec
Disk stats (read/write):
sda: ios=18/3592756, merge=40/3991, ticks=143/3408371, in_queue=3408689, util=99.84%
Run 1:
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
write_throughput: (groupid=0, jobs=8): err= 0: pid=574536: Sat Mar 5 16:59:00 2022
write: IOPS=702, BW=711MiB/s (745MB/s)(42.1GiB/60722msec); 0 zone resets
slat (usec): min=34, max=5041.6k, avg=11287.14, stdev=68759.80
clat (msec): min=67, max=8860, avg=712.02, stdev=916.11
lat (msec): min=68, max=8860, avg=723.28, stdev=926.64
clat percentiles (msec):
| 1.00th=[ 72], 5.00th=[ 75], 10.00th=[ 426], 20.00th=[ 523],
| 30.00th=[ 550], 40.00th=[ 575], 50.00th=[ 592], 60.00th=[ 617],
| 70.00th=[ 642], 80.00th=[ 667], 90.00th=[ 709], 95.00th=[ 768],
| 99.00th=[ 5805], 99.50th=[ 6879], 99.90th=[ 7953], 99.95th=[ 8221],
| 99.99th=[ 8557]
bw ( KiB/s): min=16334, max=1887969, per=100.00%, avg=768946.88, stdev=86020.37, samples=903
iops : min= 14, max= 1843, avg=750.45, stdev=84.01, samples=903
lat (msec) : 100=9.67%, 250=0.07%, 500=5.90%, 750=79.66%, 1000=2.22%
lat (msec) : 2000=0.32%, >=2000=3.34%
cpu : usr=0.67%, sys=0.62%, ctx=27604, majf=0, minf=120
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,42637,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=711MiB/s (745MB/s), 711MiB/s-711MiB/s (745MB/s-745MB/s), io=42.1GiB (45.2GB), run=60722-60722msec
Disk stats (read/write):
sda: ios=8/88672, merge=0/74, ticks=518/3528404, in_queue=3529979, util=99.87%
Run 2:
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
write_throughput: (groupid=0, jobs=8): err= 0: pid=578655: Sat Mar 5 17:07:11 2022
write: IOPS=730, BW=734MiB/s (769MB/s)(43.5GiB/60742msec); 0 zone resets
slat (usec): min=33, max=707125, avg=10879.04, stdev=25589.64
clat (msec): min=230, max=7716, avg=671.17, stdev=705.23
lat (msec): min=331, max=7716, avg=682.34, stdev=715.70
clat percentiles (msec):
| 1.00th=[ 384], 5.00th=[ 430], 10.00th=[ 460], 20.00th=[ 493],
| 30.00th=[ 518], 40.00th=[ 542], 50.00th=[ 558], 60.00th=[ 584],
| 70.00th=[ 609], 80.00th=[ 634], 90.00th=[ 676], 95.00th=[ 735],
| 99.00th=[ 5201], 99.50th=[ 6141], 99.90th=[ 7215], 99.95th=[ 7416],
| 99.99th=[ 7684]
bw ( KiB/s): min=22528, max=1310720, per=100.00%, avg=814583.35, stdev=38409.34, samples=886
iops : min= 22, max= 1280, avg=795.19, stdev=37.50, samples=886
lat (msec) : 250=0.01%, 500=22.60%, 750=73.44%, 1000=1.20%, 2000=0.52%
lat (msec) : >=2000=2.65%
cpu : usr=0.71%, sys=0.59%, ctx=29640, majf=0, minf=120
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.1%, 32=0.6%, >=64=99.3%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,44377,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=734MiB/s (769MB/s), 734MiB/s-734MiB/s (769MB/s-769MB/s), io=43.5GiB (46.7GB), run=60742-60742msec
Disk stats (read/write):
sda: ios=3/89127, merge=0/47, ticks=1438/3549874, in_queue=3552534, util=99.86%
Run 1:
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
read_iops: (groupid=0, jobs=1): err= 0: pid=572212: Sat Mar 5 16:52:56 2022
read: IOPS=55.8k, BW=218MiB/s (229MB/s)(12.8GiB/60001msec)
slat (nsec): min=850, max=5127.7k, avg=17307.11, stdev=57411.63
clat (usec): min=322, max=29725, avg=1129.04, stdev=537.24
lat (usec): min=354, max=29727, avg=1146.41, stdev=543.64
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 668], 10.00th=[ 725], 20.00th=[ 824],
| 30.00th=[ 914], 40.00th=[ 996], 50.00th=[ 1074], 60.00th=[ 1156],
| 70.00th=[ 1237], 80.00th=[ 1336], 90.00th=[ 1483], 95.00th=[ 1647],
| 99.00th=[ 2606], 99.50th=[ 3228], 99.90th=[ 9372], 99.95th=[ 9765],
| 99.99th=[13829]
bw ( KiB/s): min=201832, max=247880, per=99.99%, avg=223240.36, stdev=9235.04, samples=120
iops : min=50458, max=61970, avg=55810.10, stdev=2308.72, samples=120
lat (usec) : 500=0.13%, 750=11.88%, 1000=29.03%
lat (msec) : 2=56.68%, 4=1.95%, 10=0.29%, 20=0.04%, 50=0.01%
cpu : usr=3.21%, sys=16.85%, ctx=391933, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3348827,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=218MiB/s (229MB/s), 218MiB/s-218MiB/s (229MB/s-229MB/s), io=12.8GiB (13.7GB), run=60001-60001msec
Disk stats (read/write):
sda: ios=3458076/9, merge=3927/7, ticks=3506538/29, in_queue=3506586, util=99.88%
Run 2:
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
read_iops: (groupid=0, jobs=1): err= 0: pid=570106: Sat Mar 5 16:47:56 2022
read: IOPS=56.1k, BW=219MiB/s (230MB/s)(12.8GiB/60001msec)
slat (nsec): min=889, max=5151.8k, avg=17155.15, stdev=57727.15
clat (usec): min=248, max=33164, avg=1124.16, stdev=544.54
lat (usec): min=257, max=33165, avg=1141.39, stdev=550.88
clat percentiles (usec):
| 1.00th=[ 545], 5.00th=[ 652], 10.00th=[ 717], 20.00th=[ 816],
| 30.00th=[ 898], 40.00th=[ 988], 50.00th=[ 1074], 60.00th=[ 1156],
| 70.00th=[ 1237], 80.00th=[ 1336], 90.00th=[ 1500], 95.00th=[ 1663],
| 99.00th=[ 2606], 99.50th=[ 3195], 99.90th=[ 9372], 99.95th=[ 9896],
| 99.99th=[13960]
bw ( KiB/s): min=202376, max=236920, per=100.00%, avg=224222.93, stdev=9162.66, samples=120
iops : min=50594, max=59230, avg=56055.75, stdev=2290.67, samples=120
lat (usec) : 250=0.01%, 500=0.37%, 750=12.79%, 1000=28.84%
lat (msec) : 2=55.60%, 4=2.08%, 10=0.27%, 20=0.05%, 50=0.01%
cpu : usr=3.58%, sys=18.52%, ctx=389783, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3363453,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=219MiB/s (230MB/s), 219MiB/s-219MiB/s (230MB/s-230MB/s), io=12.8GiB (13.8GB), run=60001-60001msec
Disk stats (read/write):
sda: ios=3473045/10, merge=3948/7, ticks=3475528/27, in_queue=3475573, util=99.88%
Run 1:
read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
read_throughput: (groupid=0, jobs=8): err= 0: pid=566707: Sat Mar 5 16:41:42 2022
read: IOPS=979, BW=988MiB/s (1036MB/s)(57.0GiB/60041msec)
slat (usec): min=17, max=103694, avg=8158.16, stdev=11001.20
clat (msec): min=18, max=903, avg=512.16, stdev=94.17
lat (msec): min=30, max=914, avg=520.32, stdev=95.04
clat percentiles (msec):
| 1.00th=[ 292], 5.00th=[ 363], 10.00th=[ 401], 20.00th=[ 435],
| 30.00th=[ 464], 40.00th=[ 489], 50.00th=[ 510], 60.00th=[ 535],
| 70.00th=[ 558], 80.00th=[ 592], 90.00th=[ 634], 95.00th=[ 667],
| 99.00th=[ 743], 99.50th=[ 768], 99.90th=[ 818], 99.95th=[ 844],
| 99.99th=[ 869]
bw ( KiB/s): min=579584, max=1550336, per=99.20%, avg=1003994.18, stdev=22820.20, samples=960
iops : min= 566, max= 1514, avg=980.07, stdev=22.30, samples=960
lat (msec) : 20=0.01%, 50=0.05%, 100=0.08%, 250=0.32%, 500=45.11%
lat (msec) : 750=54.51%, 1000=0.78%
cpu : usr=0.05%, sys=0.63%, ctx=33930, majf=0, minf=123
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=58832,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=988MiB/s (1036MB/s), 988MiB/s-988MiB/s (1036MB/s-1036MB/s), io=57.0GiB (62.2GB), run=60041-60041msec
Disk stats (read/write):
sda: ios=121574/12, merge=2/7, ticks=3519202/6394, in_queue=3525674, util=99.92%
Run 2:
read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
read_throughput: (groupid=0, jobs=8): err= 0: pid=568603: Sat Mar 5 16:45:56 2022
read: IOPS=979, BW=988MiB/s (1036MB/s)(57.9GiB/60049msec)
slat (usec): min=17, max=107366, avg=8164.86, stdev=11020.93
clat (msec): min=18, max=887, avg=512.59, stdev=91.69
lat (msec): min=29, max=898, avg=520.76, stdev=92.58
clat percentiles (msec):
| 1.00th=[ 309], 5.00th=[ 368], 10.00th=[ 401], 20.00th=[ 439],
| 30.00th=[ 468], 40.00th=[ 489], 50.00th=[ 510], 60.00th=[ 535],
| 70.00th=[ 558], 80.00th=[ 592], 90.00th=[ 634], 95.00th=[ 667],
| 99.00th=[ 743], 99.50th=[ 760], 99.90th=[ 802], 99.95th=[ 818],
| 99.99th=[ 860]
bw ( KiB/s): min=608112, max=1505280, per=99.21%, avg=1003290.27, stdev=21985.68, samples=960
iops : min= 593, max= 1470, avg=979.44, stdev=21.47, samples=960
lat (msec) : 20=0.01%, 50=0.04%, 100=0.09%, 250=0.25%, 500=45.46%
lat (msec) : 750=54.30%, 1000=0.71%
cpu : usr=0.05%, sys=0.63%, ctx=33994, majf=0, minf=121
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=58793,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=988MiB/s (1036MB/s), 988MiB/s-988MiB/s (1036MB/s-1036MB/s), io=57.9GiB (62.2GB), run=60049-60049msec
Disk stats (read/write):
sda: ios=121463/14, merge=3/11, ticks=3517759/7351, in_queue=3525187, util=99.93%
Run 1:
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
write_iops: (groupid=0, jobs=1): err= 0: pid=546999: Sat Mar 5 15:58:29 2022
write: IOPS=67.2k, BW=263MiB/s (275MB/s)(15.4GiB/60001msec); 0 zone resets
slat (nsec): min=1081, max=16950k, avg=14244.89, stdev=36184.47
clat (usec): min=254, max=27114, avg=937.14, stdev=366.22
lat (usec): min=259, max=27120, avg=951.45, stdev=369.09
clat percentiles (usec):
| 1.00th=[ 553], 5.00th=[ 594], 10.00th=[ 619], 20.00th=[ 709],
| 30.00th=[ 758], 40.00th=[ 840], 50.00th=[ 906], 60.00th=[ 963],
| 70.00th=[ 1057], 80.00th=[ 1106], 90.00th=[ 1205], 95.00th=[ 1319],
| 99.00th=[ 2040], 99.50th=[ 2409], 99.90th=[ 3097], 99.95th=[ 3458],
| 99.99th=[14484]
bw ( KiB/s): min=250736, max=284808, per=99.97%, avg=268896.10, stdev=6256.64, samples=120
iops : min=62684, max=71202, avg=67223.94, stdev=1564.20, samples=120
lat (usec) : 500=0.07%, 750=28.63%, 1000=36.14%
lat (msec) : 2=34.05%, 4=1.08%, 10=0.02%, 20=0.01%, 50=0.01%
cpu : usr=4.86%, sys=33.63%, ctx=420347, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,4034790,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=263MiB/s (275MB/s), 263MiB/s-263MiB/s (275MB/s-275MB/s), io=15.4GiB (16.5GB), run=60001-60001msec
Disk stats (read/write):
sda: ios=0/4144510, merge=0/52322, ticks=0/3457248, in_queue=3457258, util=99.88%
Run 2:
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
write_iops: (groupid=0, jobs=1): err= 0: pid=547527: Sat Mar 5 16:00:08 2022
write: IOPS=66.8k, BW=261MiB/s (274MB/s)(15.3GiB/60001msec); 0 zone resets
slat (nsec): min=966, max=9306.2k, avg=14266.75, stdev=35294.48
clat (usec): min=240, max=11958, avg=943.05, stdev=300.77
lat (usec): min=245, max=11965, avg=957.38, stdev=303.12
clat percentiles (usec):
| 1.00th=[ 519], 5.00th=[ 594], 10.00th=[ 611], 20.00th=[ 717],
| 30.00th=[ 742], 40.00th=[ 840], 50.00th=[ 938], 60.00th=[ 963],
| 70.00th=[ 1074], 80.00th=[ 1123], 90.00th=[ 1221], 95.00th=[ 1336],
| 99.00th=[ 2057], 99.50th=[ 2409], 99.90th=[ 3032], 99.95th=[ 3326],
| 99.99th=[ 3949]
bw ( KiB/s): min=259776, max=282992, per=99.98%, avg=267248.71, stdev=4133.75, samples=120
iops : min=64944, max=70748, avg=66812.14, stdev=1033.45, samples=120
lat (usec) : 250=0.01%, 500=0.83%, 750=29.84%, 1000=34.28%
lat (msec) : 2=33.86%, 4=1.18%, 10=0.01%, 20=0.01%
cpu : usr=4.69%, sys=17.43%, ctx=496395, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,4009730,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=261MiB/s (274MB/s), 261MiB/s-261MiB/s (274MB/s-274MB/s), io=15.3GiB (16.4GB), run=60001-60001msec
Disk stats (read/write):
sda: ios=2/4116663, merge=0/4038, ticks=1/3430834, in_queue=3430846, util=99.85%
Run 1:
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: (groupid=0, jobs=8): err= 0: pid=548819: Sat Mar 5 16:02:27 2022
write: IOPS=984, BW=993MiB/s (1042MB/s)(58.2GiB/60044msec); 0 zone resets
slat (usec): min=32, max=78417, avg=8116.27, stdev=9462.65
clat (msec): min=16, max=778, avg=509.64, stdev=71.24
lat (msec): min=29, max=785, avg=517.75, stdev=71.80
clat percentiles (msec):
| 1.00th=[ 351], 5.00th=[ 401], 10.00th=[ 426], 20.00th=[ 456],
| 30.00th=[ 477], 40.00th=[ 493], 50.00th=[ 510], 60.00th=[ 527],
| 70.00th=[ 542], 80.00th=[ 567], 90.00th=[ 600], 95.00th=[ 625],
| 99.00th=[ 684], 99.50th=[ 701], 99.90th=[ 743], 99.95th=[ 751],
| 99.99th=[ 760]
bw ( KiB/s): min=673792, max=1343488, per=99.20%, avg=1008934.51, stdev=16791.88, samples=960
iops : min= 658, max= 1312, avg=984.78, stdev=16.41, samples=960
lat (msec) : 20=0.01%, 50=0.06%, 100=0.07%, 250=0.26%, 500=44.81%
lat (msec) : 750=55.61%, 1000=0.05%
cpu : usr=0.80%, sys=0.65%, ctx=38332, majf=0, minf=130
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,59127,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=993MiB/s (1042MB/s), 993MiB/s-993MiB/s (1042MB/s-1042MB/s), io=58.2GiB (62.5GB), run=60044-60044msec
Disk stats (read/write):
sda: ios=0/122169, merge=0/1383, ticks=0/3514331, in_queue=3514351, util=99.88%
Run 2:
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
write_throughput: (groupid=0, jobs=8): err= 0: pid=550269: Sat Mar 5 16:06:24 2022
write: IOPS=956, BW=965MiB/s (1012MB/s)(56.6GiB/60043msec); 0 zone resets
slat (usec): min=33, max=104355, avg=8354.34, stdev=10726.77
clat (msec): min=18, max=770, avg=524.32, stdev=73.06
lat (msec): min=28, max=776, avg=532.67, stdev=73.50
clat percentiles (msec):
| 1.00th=[ 355], 5.00th=[ 414], 10.00th=[ 435], 20.00th=[ 464],
| 30.00th=[ 489], 40.00th=[ 510], 50.00th=[ 527], 60.00th=[ 542],
| 70.00th=[ 567], 80.00th=[ 584], 90.00th=[ 617], 95.00th=[ 642],
| 99.00th=[ 684], 99.50th=[ 701], 99.90th=[ 726], 99.95th=[ 735],
| 99.99th=[ 760]
bw ( KiB/s): min=692040, max=1351680, per=99.18%, avg=980403.84, stdev=16475.72, samples=960
iops : min= 675, max= 1320, avg=957.14, stdev=16.09, samples=960
lat (msec) : 20=0.01%, 50=0.04%, 100=0.09%, 250=0.26%, 500=35.74%
lat (msec) : 750=64.72%, 1000=0.02%
cpu : usr=0.68%, sys=0.49%, ctx=36887, majf=0, minf=120
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,57451,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=965MiB/s (1012MB/s), 965MiB/s-965MiB/s (1012MB/s-1012MB/s), io=56.6GiB (60.8GB), run=60043-60043msec
Disk stats (read/write):
sda: ios=0/118751, merge=0/30, ticks=0/3519176, in_queue=3519199, util=99.89%
Run 1:
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
read_iops: Laying out IO file (1 file / 10240MiB)
read_iops: (groupid=0, jobs=1): err= 0: pid=551355: Sat Mar 5 16:09:36 2022
read: IOPS=59.0k, BW=234MiB/s (246MB/s)(13.7GiB/60001msec)
slat (nsec): min=878, max=873052, avg=16072.77, stdev=36953.75
clat (usec): min=301, max=6283, avg=1050.84, stdev=318.96
lat (usec): min=334, max=6389, avg=1066.97, stdev=321.31
clat percentiles (usec):
| 1.00th=[ 603], 5.00th=[ 701], 10.00th=[ 717], 20.00th=[ 783],
| 30.00th=[ 848], 40.00th=[ 955], 50.00th=[ 1004], 60.00th=[ 1090],
| 70.00th=[ 1188], 80.00th=[ 1254], 90.00th=[ 1352], 95.00th=[ 1450],
| 99.00th=[ 2278], 99.50th=[ 2638], 99.90th=[ 3359], 99.95th=[ 3621],
| 99.99th=[ 4146]
bw ( KiB/s): min=238352, max=251280, per=100.00%, avg=239868.24, stdev=1361.29, samples=120
iops : min=59588, max=62820, avg=59967.07, stdev=340.34, samples=120
lat (usec) : 500=0.01%, 750=17.95%, 1000=31.58%
lat (msec) : 2=48.81%, 4=1.64%, 10=0.02%
cpu : usr=3.60%, sys=11.69%, ctx=502117, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3598143,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=13.7GiB (14.7GB), run=60001-60001msec
Disk stats (read/write):
sda: ios=3712850/4, merge=4058/1, ticks=3528865/3, in_queue=3528869, util=99.88%
Run 2:
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
read_iops: (groupid=0, jobs=1): err= 0: pid=552005: Sat Mar 5 16:10:58 2022
read: IOPS=60.2k, BW=235MiB/s (247MB/s)(13.8GiB/60001msec)
slat (nsec): min=871, max=6631.0k, avg=15953.01, stdev=37878.40
clat (usec): min=139, max=8239, avg=1046.22, stdev=323.29
lat (usec): min=212, max=8241, avg=1062.24, stdev=325.56
clat percentiles (usec):
| 1.00th=[ 603], 5.00th=[ 652], 10.00th=[ 717], 20.00th=[ 766],
| 30.00th=[ 848], 40.00th=[ 955], 50.00th=[ 996], 60.00th=[ 1090],
| 70.00th=[ 1188], 80.00th=[ 1237], 90.00th=[ 1352], 95.00th=[ 1467],
| 99.00th=[ 2278], 99.50th=[ 2638], 99.90th=[ 3392], 99.95th=[ 3687],
| 99.99th=[ 4621]
bw ( KiB/s): min=236720, max=255072, per=100.00%, avg=240928.98, stdev=2412.99, samples=120
iops : min=59180, max=63768, avg=60232.26, stdev=603.23, samples=120
lat (usec) : 250=0.01%, 500=0.13%, 750=18.47%, 1000=31.56%
lat (msec) : 2=48.20%, 4=1.62%, 10=0.03%
cpu : usr=3.86%, sys=12.64%, ctx=498236, majf=0, minf=15
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=3614057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=13.8GiB (14.8GB), run=60001-60001msec
Disk stats (read/write):
sda: ios=3730935/14, merge=3985/9, ticks=3512795/13, in_queue=3512810, util=99.84%
Run 1:
read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: (groupid=0, jobs=8): err= 0: pid=586930: Sat Mar 5 17:21:33 2022
read: IOPS=1014, BW=1023MiB/s (1073MB/s)(59.0GiB/60038msec)
slat (usec): min=14, max=106977, avg=7879.32, stdev=10063.60
clat (msec): min=15, max=824, avg=494.71, stdev=74.71
lat (msec): min=30, max=824, avg=502.59, stdev=75.30
clat percentiles (msec):
| 1.00th=[ 317], 5.00th=[ 380], 10.00th=[ 409], 20.00th=[ 439],
| 30.00th=[ 460], 40.00th=[ 477], 50.00th=[ 493], 60.00th=[ 510],
| 70.00th=[ 527], 80.00th=[ 550], 90.00th=[ 584], 95.00th=[ 617],
| 99.00th=[ 676], 99.50th=[ 701], 99.90th=[ 743], 99.95th=[ 768],
| 99.99th=[ 785]
bw ( KiB/s): min=663400, max=1433600, per=99.22%, avg=1039631.31, stdev=17952.69, samples=960
iops : min= 647, max= 1400, avg=1014.97, stdev=17.53, samples=960
lat (msec) : 20=0.01%, 50=0.04%, 100=0.08%, 250=0.27%, 500=54.58%
lat (msec) : 750=45.77%, 1000=0.08%
cpu : usr=0.06%, sys=0.61%, ctx=35514, majf=0, minf=124
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=60922,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1023MiB/s (1073MB/s), 1023MiB/s-1023MiB/s (1073MB/s-1073MB/s), io=59.0GiB (64.4GB), run=60038-60038msec
Disk stats (read/write):
sda: ios=125871/4, merge=0/3, ticks=3525405/80, in_queue=3525513, util=99.93%
Run 2:
read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 8 processes
read_throughput: (groupid=0, jobs=8): err= 0: pid=588421: Sat Mar 5 17:23:40 2022
read: IOPS=1013, BW=1022MiB/s (1072MB/s)(59.9GiB/60049msec)
slat (usec): min=16, max=103922, avg=7885.94, stdev=9751.39
clat (msec): min=16, max=800, avg=495.07, stdev=70.68
lat (msec): min=30, max=800, avg=502.95, stdev=71.22
clat percentiles (msec):
| 1.00th=[ 342], 5.00th=[ 388], 10.00th=[ 414], 20.00th=[ 439],
| 30.00th=[ 460], 40.00th=[ 477], 50.00th=[ 493], 60.00th=[ 510],
| 70.00th=[ 531], 80.00th=[ 550], 90.00th=[ 584], 95.00th=[ 609],
| 99.00th=[ 667], 99.50th=[ 693], 99.90th=[ 726], 99.95th=[ 743],
| 99.99th=[ 776]
bw ( KiB/s): min=698368, max=1388544, per=99.24%, avg=1038770.43, stdev=16903.94, samples=960
iops : min= 682, max= 1356, avg=1014.11, stdev=16.51, samples=960
lat (msec) : 20=0.01%, 50=0.05%, 100=0.09%, 250=0.24%, 500=54.19%
lat (msec) : 750=46.22%, 1000=0.04%
cpu : usr=0.06%, sys=0.58%, ctx=36212, majf=0, minf=120
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=60872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1022MiB/s (1072MB/s), 1022MiB/s-1022MiB/s (1072MB/s-1072MB/s), io=59.9GiB (64.4GB), run=60049-60049msec
Disk stats (read/write):
sda: ios=125755/9, merge=0/7, ticks=3526714/199, in_queue=3526971, util=99.93%