Skip to content

Instantly share code, notes, and snippets.

@danielmai

danielmai/1_summary.md

Last active January 11, 2024 01:32
Show Gist options
  • Save danielmai/a76105a11ff4c2e34df2e31baa678c50 to your computer and use it in GitHub Desktop.
Save danielmai/a76105a11ff4c2e34df2e31baa678c50 to your computer and use it in GitHub Desktop.
Download ZIP
fio tests on aws instances
Raw
1_summary.md

Summary: Fio tests

r5.2xlarge

r5.2xlarge instance in eu-west-1 with 200 GB EBS io1 volume and 3000 IOPS.

IOPS 95-%tile (ms) 99-%tile (ms) Rate (MB/s)
Random read 3009 343 424 12
Random write 14,969 407 1,433 59
Sequential read 107,215 35 37 428
Sequential write 118,068 51 51 472

i3.2xlarge

i3.2xlarge instance in eu-west-1 using mounted locally-attached (instance store) NVMe SSD.

IOPS 95-%tile (ms) 99-%tile (ms) Rate (MB/s)
Random read 178,119 8.7 9.7 712
Random write 199,107 54.5 55.0 796
Sequential read 470,741 7.7 10.9 1838.9
Sequential write 209,518 54 55 838

r5d.2xlarge

r5d.2xlarge instance in eu-west-1 with the 300 GB locally attached (instance store) NVMe SSD.

IOPS 95-%tile (ms) 99-%tile (ms) Rate (MB/s)
Random read 117,790 9.1 9.5 471
Random write 75,798 55.5 55.5 303
Sequential read 132,822 27.5 30.3 531
Sequential write 209,518 55.5 55.5 304
Raw
fio-i3.2xlarge.md

Summary

i3.2xlarge instance in eu-west-1 using mounted locally-attached NVMe SSD.

IOPS 95-%tile 99-%tile Rate
Random read 178,119 8.7 ms 9.7 ms 712 MB/s
Random write 199,107 54.5 ms 55.0 ms 796 MB/s
Sequential read 470,741 7.7 ms 10.9 ms 1838.9MB/s
Sequential write 209,518 54 ms 55 ms 838 MB/s

Fio tests

Random reads

fiotest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)

fiotest: (groupid=0, jobs=32): err= 0: pid=3811: Mon May  6 23:43:01 2019
  read : io=131072MB, bw=712476KB/s, iops=178119, runt=188382msec
    slat (usec): min=79, max=24712, avg=175.37, stdev=229.99
    clat (usec): min=1, max=36187, avg=5561.46, stdev=1470.65
     lat (usec): min=106, max=36341, avg=5737.45, stdev=1493.08
    clat percentiles (usec):
     |  1.00th=[ 4512],  5.00th=[ 4704], 10.00th=[ 4768], 20.00th=[ 4896],
     | 30.00th=[ 4960], 40.00th=[ 5024], 50.00th=[ 5088], 60.00th=[ 5216],
     | 70.00th=[ 5280], 80.00th=[ 5472], 90.00th=[ 7712], 95.00th=[ 8768],
     | 99.00th=[ 9792], 99.50th=[11584], 99.90th=[21120], 99.95th=[23424],
     | 99.99th=[27008]
    bw (KB  /s): min=19617, max=24752, per=3.13%, avg=22304.09, stdev=502.88
    lat (usec) : 2=0.01%, 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.04%, 10=99.17%, 20=0.65%, 50=0.15%
  cpu          : usr=3.32%, sys=9.60%, ctx=33644502, majf=0, minf=1229
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=33554432/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=131072MB, aggrb=712476KB/s, minb=712476KB/s, maxb=712476KB/s, mint=188382msec, maxt=188382msec

Disk stats (read/write):
  nvme0n1: ios=33550982/69354, merge=0/263, ticks=4710812/0, in_queue=4665308, util=100.00%

Random writes

fiotest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes

fiotest: (groupid=0, jobs=32): err= 0: pid=3851: Mon May  6 23:46:03 2019
  write: io=131072MB, bw=796431KB/s, iops=199107, runt=168524msec
    slat (usec): min=2, max=247824, avg=157.47, stdev=2875.65
    clat (usec): min=1, max=302141, avg=4931.55, stdev=15345.96
     lat (usec): min=4, max=302146, avg=5089.14, stdev=15567.23
    clat percentiles (usec):
     |  1.00th=[  121],  5.00th=[  127], 10.00th=[  131], 20.00th=[  133],
     | 30.00th=[  137], 40.00th=[  173], 50.00th=[  183], 60.00th=[  185],
     | 70.00th=[  189], 80.00th=[  191], 90.00th=[  215], 95.00th=[54528],
     | 99.00th=[55040], 99.50th=[58624], 99.90th=[66048], 99.95th=[70144],
     | 99.99th=[76288]
    bw (KB  /s): min=19940, max=400328, per=3.16%, avg=25183.26, stdev=16041.16
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=90.43%, 500=0.38%, 750=0.08%, 1000=0.03%
    lat (msec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.15%, 50=0.37%
    lat (msec) : 100=8.47%, 250=0.01%, 500=0.01%
  cpu          : usr=0.82%, sys=2.60%, ctx=99261, majf=0, minf=216
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=33554432/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=131072MB, aggrb=796430KB/s, minb=796430KB/s, maxb=796430KB/s, mint=168524msec, maxt=168524msec

Disk stats (read/write):
  nvme0n1: ios=33/21480524, merge=0/217, ticks=8/45944184, in_queue=45681848, util=95.19%

Sequential reads

fiotest: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes

fiotest: (groupid=0, jobs=32): err= 0: pid=3897: Mon May  6 23:47:34 2019
  read : io=131072MB, bw=1838.9MB/s, iops=470741, runt= 71280msec
    slat (usec): min=0, max=17189, avg=66.92, stdev=661.03
    clat (usec): min=1, max=18559, avg=2097.58, stdev=3088.98
     lat (usec): min=2, max=27433, avg=2164.55, stdev=3116.39
    clat percentiles (usec):
     |  1.00th=[   54],  5.00th=[   55], 10.00th=[   55], 20.00th=[   56],
     | 30.00th=[   58], 40.00th=[   65], 50.00th=[   81], 60.00th=[  668],
     | 70.00th=[ 1864], 80.00th=[ 5984], 90.00th=[ 7392], 95.00th=[ 7776],
     | 99.00th=[10944], 99.50th=[11968], 99.90th=[13760], 99.95th=[14400],
     | 99.99th=[15808]
    bw (KB  /s): min=18872, max=119024, per=3.13%, avg=58916.20, stdev=2854.82
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=51.44%, 250=2.03%, 500=4.20%, 750=3.46%, 1000=2.75%
    lat (msec) : 2=6.74%, 4=4.75%, 10=22.88%, 20=1.75%
  cpu          : usr=0.28%, sys=3.81%, ctx=521366, majf=0, minf=1263
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=33554432/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=131072MB, aggrb=1838.9MB/s, minb=1838.9MB/s, maxb=1838.9MB/s, mint=71280msec, maxt=71280msec

Disk stats (read/write):
  nvme0n1: ios=1045630/12227, merge=0/37, ticks=8869176/48508, in_queue=8897792, util=97.00%

Sequential writes

fiotest: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes

fiotest: (groupid=0, jobs=32): err= 0: pid=3936: Mon May  6 23:50:23 2019
  write: io=131072MB, bw=838075KB/s, iops=209518, runt=160150msec
    slat (usec): min=1, max=85020, avg=149.92, stdev=2770.69
    clat (usec): min=1, max=125859, avg=4688.32, stdev=14792.08
     lat (usec): min=3, max=125864, avg=4838.38, stdev=15005.93
    clat percentiles (usec):
     |  1.00th=[  104],  5.00th=[  111], 10.00th=[  113], 20.00th=[  115],
     | 30.00th=[  117], 40.00th=[  129], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  169], 80.00th=[  173], 90.00th=[  229], 95.00th=[54016],
     | 99.00th=[55040], 99.50th=[55040], 99.90th=[56064], 99.95th=[59136],
     | 99.99th=[68096]
    bw (KB  /s): min=21829, max=245592, per=3.16%, avg=26491.98, stdev=16337.53
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
    lat (usec) : 100=0.01%, 250=90.45%, 500=0.78%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.02%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.55%
    lat (msec) : 100=8.12%, 250=0.01%
  cpu          : usr=0.75%, sys=2.41%, ctx=96043, majf=0, minf=275
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=33554432/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=131072MB, aggrb=838075KB/s, minb=838075KB/s, maxb=838075KB/s, mint=160150msec, maxt=160150msec

Disk stats (read/write):
  nvme0n1: ios=32/972433, merge=0/96, ticks=0/87582548, in_queue=87630236, util=98.13%
Raw
fio-r5.2xlarge.md

Summary

r5.2xlarge instance in eu-west-1 with 200 GB EBS io1 volume and 3000 IOPS.

IOPS 95-%tile 99-%tile Rate
Random read 3009 343 ms 424 ms 12 MB/s
Random write 14,969 407 ms 1,433 ms 59 MB/s
Sequential read 107,215 35 ms 37 ms 428 MB/s
Sequential write 118,068 51 ms 51 ms 472 MB/s

Fio tests

Random reads

fiotest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fiotest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: (groupid=0, jobs=32): err= 0: pid=2680: Mon May  6 23:16:35 2019
  read : io=2821.7MB, bw=12039KB/s, iops=3009, runt=240011msec
    slat (usec): min=161, max=287965, avg=10630.40, stdev=5026.83
    clat (usec): min=1, max=1727.6K, avg=329586.61, stdev=45329.72
     lat (msec): min=8, max=1727, avg=340.22, stdev=46.44
    clat percentiles (msec):
     |  1.00th=[  151],  5.00th=[  318], 10.00th=[  326], 20.00th=[  326],
     | 30.00th=[  330], 40.00th=[  330], 50.00th=[  330], 60.00th=[  330],
     | 70.00th=[  330], 80.00th=[  334], 90.00th=[  338], 95.00th=[  347],
     | 99.00th=[  424], 99.50th=[  529], 99.90th=[  848], 99.95th=[  979],
     | 99.99th=[ 1287]
    bw (KB  /s): min=   30, max= 1680, per=3.12%, avg=375.93, stdev=51.60
    lat (usec) : 2=0.01%, 4=0.01%
    lat (msec) : 10=0.07%, 20=0.20%, 50=0.36%, 100=0.17%, 250=0.93%
    lat (msec) : 500=97.70%, 750=0.42%, 1000=0.11%, 2000=0.05%
  cpu          : usr=0.02%, sys=0.09%, ctx=723036, majf=0, minf=1190
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=722346/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=2821.7MB, aggrb=12038KB/s, minb=12038KB/s, maxb=12038KB/s, mint=240011msec, maxt=240011msec

Disk stats (read/write):
  nvme0n1: ios=722282/28, merge=0/62, ticks=7672460/304, in_queue=7432112, util=99.02%

Random writes

3
fiotest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fiotest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes
fiotest: (groupid=0, jobs=32): err= 0: pid=2720: Mon May  6 23:20:38 2019
  write: io=14043MB, bw=59877KB/s, iops=14969, runt=240151msec
    slat (usec): min=2, max=207960, avg=2132.87, stdev=20613.86
    clat (usec): min=3, max=2447.7K, avg=66233.40, stdev=252516.10
     lat (usec): min=109, max=2447.8K, avg=68366.51, stdev=260196.10
    clat percentiles (usec):
     |  1.00th=[  125],  5.00th=[  157], 10.00th=[  159], 20.00th=[  159],
     | 30.00th=[  161], 40.00th=[  163], 50.00th=[  163], 60.00th=[  165],
     | 70.00th=[  167], 80.00th=[  173], 90.00th=[203776], 95.00th=[407552],
     | 99.00th=[1433600], 99.50th=[1630208], 99.90th=[2039808], 99.95th=[2244608],
     | 99.99th=[2441216]
    bw (KB  /s): min=   45, max=451936, per=3.67%, avg=2197.83, stdev=18641.47
    lat (usec) : 4=0.01%, 10=0.01%, 250=85.46%, 500=0.03%, 750=0.01%
    lat (usec) : 1000=0.02%
    lat (msec) : 2=0.13%, 4=0.29%, 10=0.39%, 20=0.45%, 50=1.91%
    lat (msec) : 100=0.18%, 250=4.02%, 500=4.27%, 750=0.20%, 1000=0.07%
    lat (msec) : 2000=2.37%, >=2000=0.21%
  cpu          : usr=0.06%, sys=0.20%, ctx=42207, majf=0, minf=232
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=3594880/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=14043MB, aggrb=59876KB/s, minb=59876KB/s, maxb=59876KB/s, mint=240151msec, maxt=240151msec

Disk stats (read/write):
  nvme0n1: ios=592/719993, merge=0/155, ticks=2844/3921248, in_queue=3681092, util=98.56%

Sequential reads

fiotest: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes
Jobs: 32 (f=32): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [100.0% done] [417.8MB/0KB/0KB /s] [107K/0/0 iops] [eta 00m:00s]       
fiotest: (groupid=0, jobs=32): err= 0: pid=2822: Mon May  6 23:38:19 2019
  read : io=100528MB, bw=428860KB/s, iops=107215, runt=240034msec
    slat (usec): min=0, max=226022, avg=297.08, stdev=2860.08
    clat (usec): min=2, max=226099, avg=9231.19, stdev=13124.70
     lat (usec): min=3, max=226101, avg=9528.32, stdev=13229.30
    clat percentiles (usec):
     |  1.00th=[   49],  5.00th=[   50], 10.00th=[   51], 20.00th=[   54],
     | 30.00th=[   57], 40.00th=[   61], 50.00th=[   73], 60.00th=[ 4128],
     | 70.00th=[11328], 80.00th=[24704], 90.00th=[33024], 95.00th=[35584],
     | 99.00th=[37632], 99.50th=[38656], 99.90th=[46336], 99.95th=[58624],
     | 99.99th=[91648]
    bw (KB  /s): min=    5, max=130027, per=3.13%, avg=13419.90, stdev=1950.03
    lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=2.34%, 100=49.86%
    lat (usec) : 250=0.14%, 500=0.31%, 750=0.61%, 1000=0.55%
    lat (msec) : 2=2.29%, 4=3.71%, 10=8.84%, 20=7.94%, 50=23.34%
    lat (msec) : 100=0.07%, 250=0.01%
  cpu          : usr=0.21%, sys=0.66%, ctx=483553, majf=0, minf=1206
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=25735254/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=100528MB, aggrb=428860KB/s, minb=428860KB/s, maxb=428860KB/s, mint=240034msec, maxt=240034msec

Disk stats (read/write):
  nvme0n1: ios=402181/17, merge=0/7, ticks=14564276/660, in_queue=14348384, util=98.93%

Sequential writes

fiotest: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes
Jobs: 32 (f=32): [WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW] [100.0% done] [0KB/418.2MB/0KB /s] [0/107K/0 iops] [eta 00m:00s]
fiotest: (groupid=0, jobs=32): err= 0: pid=2863: Mon May  6 23:42:26 2019
  write: io=110711MB, bw=472275KB/s, iops=118068, runt=240047msec
    slat (usec): min=1, max=231851, avg=269.87, stdev=3736.86
    clat (usec): min=2, max=430112, avg=8400.33, stdev=19265.88
     lat (usec): min=89, max=430117, avg=8670.29, stdev=19523.09
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  101], 10.00th=[  103], 20.00th=[  107],
     | 30.00th=[  117], 40.00th=[  135], 50.00th=[  139], 60.00th=[  143],
     | 70.00th=[  145], 80.00th=[  155], 90.00th=[47360], 95.00th=[51456],
     | 99.00th=[51456], 99.50th=[51456], 99.90th=[199680], 99.95th=[203776],
     | 99.99th=[214016]
    bw (KB  /s): min= 1383, max=400464, per=3.12%, avg=14745.51, stdev=14908.07
    lat (usec) : 4=0.01%, 10=0.01%, 100=3.40%, 250=79.59%, 500=0.12%
    lat (usec) : 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.03%, 50=11.28%
    lat (msec) : 100=5.44%, 250=0.11%, 500=0.01%
  cpu          : usr=0.42%, sys=1.14%, ctx=156948, majf=0, minf=250
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=28342060/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=110711MB, aggrb=472275KB/s, minb=472275KB/s, maxb=472275KB/s, mint=240047msec, maxt=240047msec

Disk stats (read/write):
  nvme0n1: ios=252/400674, merge=0/146, ticks=1604/3877704, in_queue=3625788, util=98.23%
Raw
fio-r5d.2xlarge.md

Summary

r5d.2xlarge instance in eu-west-1 with the 300 GB locally attached NVMe SSD.

IOPS 95-%tile (ms) 99-%tile (ms) Rate (MB/s)
Random read 117,790 9.1 9.5 471
Random write 75,798 55.5 55.5 303
Sequential read 132,822 27.5 30.3 531
Sequential write 209,518 55.5 55.5 304

Fio tests

Random reads

===> Running randread test
3
fiotest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fiotest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)
fiotest: Laying out IO file(s) (1 file(s) / 4096MB)

fiotest: (groupid=0, jobs=32): err= 0: pid=2675: Tue May  7 01:21:02 2019
  read : io=110429MB, bw=471161KB/s, iops=117790, runt=240001msec
    slat (usec): min=67, max=17758, avg=269.64, stdev=75.81
    clat (usec): min=1, max=26557, avg=8422.45, stdev=507.14
     lat (usec): min=82, max=26779, avg=8692.42, stdev=518.18
    clat percentiles (usec):
     |  1.00th=[ 6240],  5.00th=[ 7776], 10.00th=[ 7968], 20.00th=[ 8160],
     | 30.00th=[ 8256], 40.00th=[ 8384], 50.00th=[ 8384], 60.00th=[ 8512],
     | 70.00th=[ 8640], 80.00th=[ 8768], 90.00th=[ 8896], 95.00th=[ 9152],
     | 99.00th=[ 9536], 99.50th=[ 9792], 99.90th=[10688], 99.95th=[11072],
     | 99.99th=[11840]
    bw (KB  /s): min=14072, max=19984, per=3.13%, avg=14727.66, stdev=603.63
    lat (usec) : 2=0.01%, 4=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
    lat (usec) : 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=99.69%, 20=0.31%, 50=0.01%
  cpu          : usr=0.89%, sys=3.32%, ctx=28271543, majf=0, minf=1194
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=28269759/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=110429MB, aggrb=471160KB/s, minb=471160KB/s, maxb=471160KB/s, mint=240001msec, maxt=240001msec

Disk stats (read/write):
  nvme1n1: ios=28265317/12, merge=0/38, ticks=7350468/0, in_queue=7476952, util=100.00%

Random writes

===> Running randwrite test
3
fiotest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fiotest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes

fiotest: (groupid=0, jobs=32): err= 0: pid=2725: Tue May  7 01:25:15 2019
  write: io=71076MB, bw=303195KB/s, iops=75798, runt=240048msec
    slat (usec): min=2, max=126990, avg=420.43, stdev=4766.21
    clat (usec): min=2, max=127123, avg=13083.77, stdev=23299.89
     lat (usec): min=108, max=127126, avg=13504.31, stdev=23554.79
    clat percentiles (usec):
     |  1.00th=[  112],  5.00th=[  120], 10.00th=[  123], 20.00th=[  129],
     | 30.00th=[  147], 40.00th=[  157], 50.00th=[  159], 60.00th=[  163],
     | 70.00th=[  169], 80.00th=[51456], 90.00th=[55552], 95.00th=[55552],
     | 99.00th=[55552], 99.50th=[63744], 99.90th=[98816], 99.95th=[107008],
     | 99.99th=[122368]
    bw (KB  /s): min= 6134, max=439556, per=3.14%, avg=9513.36, stdev=16727.47
    lat (usec) : 4=0.01%, 250=75.78%, 500=0.08%, 750=0.01%, 1000=0.02%
    lat (msec) : 2=0.07%, 4=0.04%, 10=0.03%, 20=0.09%, 50=3.20%
    lat (msec) : 100=20.61%, 250=0.08%
  cpu          : usr=0.33%, sys=0.82%, ctx=145167, majf=0, minf=245
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=18195343/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=71076MB, aggrb=303195KB/s, minb=303195KB/s, maxb=303195KB/s, mint=240048msec, maxt=240048msec

Disk stats (read/write):
  nvme1n1: ios=32/11562379, merge=0/144, ticks=4/15623184, in_queue=15375948, util=98.51%

Sequential reads

===> Running read test
3
fiotest: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fiotest: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes

fiotest: (groupid=0, jobs=32): err= 0: pid=2837: Tue May  7 01:30:00 2019
  read : io=124535MB, bw=531289KB/s, iops=132822, runt=240027msec
    slat (usec): min=0, max=32492, avg=239.83, stdev=2174.21
    clat (usec): min=1, max=32564, avg=7455.19, stdev=9676.70
     lat (usec): min=53, max=32565, avg=7695.07, stdev=9738.48
    clat percentiles (usec):
     |  1.00th=[   54],  5.00th=[   55], 10.00th=[   55], 20.00th=[   57],
     | 30.00th=[   58], 40.00th=[   60], 50.00th=[   66], 60.00th=[ 5984],
     | 70.00th=[12480], 80.00th=[17536], 90.00th=[23424], 95.00th=[27520],
     | 99.00th=[30336], 99.50th=[30336], 99.90th=[30592], 99.95th=[30592],
     | 99.99th=[30848]
    bw (KB  /s): min=    6, max=222448, per=3.13%, avg=16620.20, stdev=2996.30
    lat (usec) : 2=0.01%, 4=0.01%, 100=51.57%, 250=0.14%, 500=0.84%
    lat (usec) : 750=0.38%, 1000=0.68%
    lat (msec) : 2=1.45%, 4=2.83%, 10=7.45%, 20=19.86%, 50=14.80%
  cpu          : usr=0.24%, sys=0.79%, ctx=501031, majf=0, minf=1202
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=31880928/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=124535MB, aggrb=531289KB/s, minb=531289KB/s, maxb=531289KB/s, mint=240027msec, maxt=240027msec

Disk stats (read/write):
  nvme1n1: ios=995741/5, merge=0/5, ticks=30367552/0, in_queue=30207272, util=99.10%

Sequential writes

fiotest: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.1.5
Starting 32 processes
Jobs: 32 (f=32): [WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW] [100.0% done] [0KB/418.2MB/0KB /s] [0/107K/0 iops] [eta 00m:00s]
fiotest: (groupid=0, jobs=32): err= 0: pid=2863: Mon May  6 23:42:26 2019
  write: io=110711MB, bw=472275KB/s, iops=118068, runt=240047msec
    slat (usec): min=1, max=231851, avg=269.87, stdev=3736.86
    clat (usec): min=2, max=430112, avg=8400.33, stdev=19265.88
     lat (usec): min=89, max=430117, avg=8670.29, stdev=19523.09
    clat percentiles (usec):
     |  1.00th=[   92],  5.00th=[  101], 10.00th=[  103], 20.00th=[  107],
     | 30.00th=[  117], 40.00th=[  135], 50.00th=[  139], 60.00th=[  143],
     | 70.00th=[  145], 80.00th=[  155], 90.00th=[47360], 95.00th=[51456],
     | 99.00th=[51456], 99.50th=[51456], 99.90th=[199680], 99.95th=[203776],
     | 99.99th=[214016]
    bw (KB  /s): min= 1383, max=400464, per=3.12%, avg=14745.51, stdev=14908.07
    lat (usec) : 4=0.01%, 10=0.01%, 100=3.40%, 250=79.59%, 500=0.12%
    lat (usec) : 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.03%, 50=11.28%
    lat (msec) : 100=5.44%, 250=0.11%, 500=0.01%
  cpu          : usr=0.42%, sys=1.14%, ctx=156948, majf=0, minf=250
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=28342060/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=110711MB, aggrb=472275KB/s, minb=472275KB/s, maxb=472275KB/s, mint=240047msec, maxt=240047msec

Disk stats (read/write):
  nvme0n1: ios=252/400674, merge=0/146, ticks=1604/3877704, in_queue=3625788, util=98.23%
Raw
fio.sh
#!/bin/bash
# Set DISK_DEV to the specific disk device to test.
DISK_DEV=${1:-/dev/nvme0n1p1}
for rw in "randread" "randwrite" "read" "write"; do
# clear disk caches
echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo blockdev --flushbufs $DISK_DEV
fio --name=fiotest --ioengine=libaio --iodepth=32 --rw=$rw --bs=4k --direct=0 --size=4G --numjobs=32 --runtime=240 --group_reporting
done | tee fio.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment