SSD bench recordsize = 4K

fio-2.2.12-10-gb9c8
Starting 16 processes
seq-read: Laying out IO file(s) (1 file(s) / 8192MB)
Jobs: 4 (f=4): [_(12),w(4)] [37.7% done] [0KB/81380KB/0KB /s] [0/20.4K/0 iops] [eta 05m:48s]
seq-read: (groupid=0, jobs=4): err= 0: pid=41643: Wed Dec  2 12:43:00 2015
  read : io=32768MB, bw=779882KB/s, iops=194970, runt= 43025msec
    slat (usec): min=0, max=659, avg= 4.40, stdev= 5.99
    clat (usec): min=4, max=17736, avg=625.58, stdev=308.24
     lat (usec): min=11, max=17739, avg=629.98, stdev=307.95
    clat percentiles (usec):
     |  1.00th=[  159],  5.00th=[  221], 10.00th=[  278], 20.00th=[  370],
     | 30.00th=[  446], 40.00th=[  516], 50.00th=[  580], 60.00th=[  652],
     | 70.00th=[  732], 80.00th=[  844], 90.00th=[ 1020], 95.00th=[ 1192],
     | 99.00th=[ 1560], 99.50th=[ 1736], 99.90th=[ 2192], 99.95th=[ 2512],
     | 99.99th=[ 3792]
    bw (KB  /s): min=150280, max=237856, per=25.20%, avg=196509.50, stdev=16451.82
    lat (usec) : 10=0.01%, 20=0.01%, 50=0.03%, 100=0.18%, 250=7.21%
    lat (usec) : 500=30.51%, 750=33.68%, 1000=17.48%
    lat (msec) : 2=10.72%, 4=0.17%, 10=0.01%, 20=0.01%
  cpu          : usr=40.69%, sys=200.29%, ctx=20340554, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.6%, 16=74.9%, 32=24.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.6%, 8=4.3%, 16=1.9%, 32=0.2%, 64=0.0%, >=64=0.0%
     issued    : total=r=8388608/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand-read: (groupid=1, jobs=4): err= 0: pid=41717: Wed Dec  2 12:43:00 2015
  read : io=16103MB, bw=274812KB/s, iops=68702, runt= 60002msec
    slat (usec): min=0, max=856, avg= 3.99, stdev= 4.46
    clat (usec): min=18, max=25688, avg=1841.70, stdev=915.25
     lat (usec): min=21, max=25691, avg=1845.68, stdev=915.08
    clat percentiles (usec):
     |  1.00th=[  498],  5.00th=[  724], 10.00th=[  900], 20.00th=[ 1144],
     | 30.00th=[ 1368], 40.00th=[ 1544], 50.00th=[ 1736], 60.00th=[ 1928],
     | 70.00th=[ 2128], 80.00th=[ 2384], 90.00th=[ 2832], 95.00th=[ 3280],
     | 99.00th=[ 4576], 99.50th=[ 5664], 99.90th=[10176], 99.95th=[12096],
     | 99.99th=[15936]
    bw (KB  /s): min=60984, max=73144, per=25.01%, avg=68717.77, stdev=2735.90
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=1.04%
    lat (usec) : 750=4.58%, 1000=8.10%
    lat (msec) : 2=50.28%, 4=34.19%, 10=1.70%, 20=0.11%, 50=0.01%
  cpu          : usr=17.23%, sys=101.88%, ctx=18222729, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=71.4%, 32=28.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.9%, 8=4.3%, 16=1.7%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=4122317/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq-write: (groupid=2, jobs=4): err= 0: pid=41749: Wed Dec  2 12:43:00 2015
  write: io=32768MB, bw=714487KB/s, iops=178621, runt= 46963msec
    slat (usec): min=0, max=9870, avg=11.24, stdev=48.78
    clat (usec): min=1, max=23622, avg=596.69, stdev=816.49
     lat (usec): min=20, max=23625, avg=607.93, stdev=815.71
    clat percentiles (usec):
     |  1.00th=[   29],  5.00th=[   65], 10.00th=[   94], 20.00th=[  151],
     | 30.00th=[  211], 40.00th=[  282], 50.00th=[  370], 60.00th=[  474],
     | 70.00th=[  604], 80.00th=[  796], 90.00th=[ 1160], 95.00th=[ 1720],
     | 99.00th=[ 4832], 99.50th=[ 5536], 99.90th=[ 7008], 99.95th=[ 7840],
     | 99.99th=[10304]
    bw (KB  /s): min=153689, max=217984, per=25.00%, avg=178655.95, stdev=11137.99
    lat (usec) : 2=0.01%, 4=0.02%, 10=0.17%, 20=0.42%, 50=2.19%
    lat (usec) : 100=8.19%, 250=24.58%, 500=26.56%, 750=15.74%, 1000=8.75%
    lat (msec) : 2=9.35%, 4=2.37%, 10=1.67%, 20=0.01%, 50=0.01%
  cpu          : usr=35.35%, sys=644.33%, ctx=13666014, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.1%, 8=8.6%, 16=78.7%, 32=11.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.7%, 8=2.2%, 16=2.3%, 32=1.8%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8388608/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand-write: (groupid=3, jobs=4): err= 0: pid=41770: Wed Dec  2 12:43:00 2015
  write: io=4937.3MB, bw=84240KB/s, iops=21060, runt= 60016msec
    slat (usec): min=0, max=1507, avg= 4.62, stdev= 7.78
    clat (usec): min=1, max=77725, avg=6053.69, stdev=7108.68
     lat (usec): min=19, max=77728, avg=6058.31, stdev=7108.55
    clat percentiles (usec):
     |  1.00th=[   31],  5.00th=[   87], 10.00th=[  434], 20.00th=[  812],
     | 30.00th=[ 1256], 40.00th=[ 1912], 50.00th=[ 2672], 60.00th=[ 4256],
     | 70.00th=[ 7328], 80.00th=[11456], 90.00th=[17024], 95.00th=[21120],
     | 99.00th=[28800], 99.50th=[32384], 99.90th=[39680], 99.95th=[43264],
     | 99.99th=[50944]
    bw (KB  /s): min=17512, max=24712, per=25.01%, avg=21070.20, stdev=873.82
    lat (usec) : 2=0.01%, 4=0.04%, 10=0.03%, 20=0.05%, 50=2.94%
    lat (usec) : 100=2.31%, 250=2.42%, 500=3.57%, 750=7.03%, 1000=6.39%
    lat (msec) : 2=16.38%, 4=17.54%, 10=18.12%, 20=17.18%, 50=6.00%
    lat (msec) : 100=0.01%
  cpu          : usr=5.56%, sys=41.14%, ctx=4634585, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=71.5%, 32=28.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.9%, 8=4.4%, 16=1.6%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1263944/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=32768MB, aggrb=779882KB/s, minb=779882KB/s, maxb=779882KB/s, mint=43025msec, maxt=43025msec

Run status group 1 (all jobs):
   READ: io=16103MB, aggrb=274811KB/s, minb=274811KB/s, maxb=274811KB/s, mint=60002msec, maxt=60002msec

Run status group 2 (all jobs):
  WRITE: io=32768MB, aggrb=714486KB/s, minb=714486KB/s, maxb=714486KB/s, mint=46963msec, maxt=46963msec

Run status group 3 (all jobs):
  WRITE: io=4937.3MB, aggrb=84240KB/s, minb=84240KB/s, maxb=84240KB/s, mint=60016msec, maxt=60016msec
[root@build_zone_71 ~/fio]#

Completed 3 runs all with similar figures here is the 3rd result for ref

Starting 16 processes
Jobs: 4 (f=4): [_(12),w(4)] [38.3% done] [0KB/84712KB/0KB /s] [0/21.2K/0 iops] [eta 05m:45s]
seq-read: (groupid=0, jobs=4): err= 0: pid=48562: Wed Dec  2 13:01:17 2015
  read : io=32768MB, bw=729080KB/s, iops=182269, runt= 46023msec
    slat (usec): min=0, max=509, avg= 4.71, stdev= 7.44
    clat (usec): min=2, max=7160, avg=669.91, stdev=431.61
     lat (usec): min=11, max=7162, avg=674.62, stdev=431.34
    clat percentiles (usec):
     |  1.00th=[  120],  5.00th=[  173], 10.00th=[  201], 20.00th=[  274],
     | 30.00th=[  370], 40.00th=[  482], 50.00th=[  588], 60.00th=[  700],
     | 70.00th=[  836], 80.00th=[ 1004], 90.00th=[ 1272], 95.00th=[ 1480],
     | 99.00th=[ 1960], 99.50th=[ 2192], 99.90th=[ 2800], 99.95th=[ 3120],
     | 99.99th=[ 3824]
    bw (KB  /s): min=164912, max=243328, per=25.19%, avg=183649.32, stdev=13031.26
    lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.11%, 100=0.50%
    lat (usec) : 250=16.72%, 500=24.46%, 750=22.09%, 1000=15.88%
    lat (msec) : 2=19.34%, 4=0.88%, 10=0.01%
  cpu          : usr=38.88%, sys=179.76%, ctx=21134126, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.4%, 16=74.1%, 32=25.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.7%, 8=4.3%, 16=1.9%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=8388608/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand-read: (groupid=1, jobs=4): err= 0: pid=48629: Wed Dec  2 13:01:17 2015
  read : io=15915MB, bw=271594KB/s, iops=67898, runt= 60003msec
    slat (usec): min=0, max=390, avg= 3.92, stdev= 4.39
    clat (usec): min=17, max=11508, avg=1864.05, stdev=869.35
     lat (usec): min=22, max=11510, avg=1867.97, stdev=869.17
    clat percentiles (usec):
     |  1.00th=[  482],  5.00th=[  676], 10.00th=[  836], 20.00th=[ 1080],
     | 30.00th=[ 1320], 40.00th=[ 1560], 50.00th=[ 1784], 60.00th=[ 2008],
     | 70.00th=[ 2224], 80.00th=[ 2480], 90.00th=[ 2960], 95.00th=[ 3472],
     | 99.00th=[ 4448], 99.50th=[ 4960], 99.90th=[ 5984], 99.95th=[ 6432],
     | 99.99th=[ 7456]
    bw (KB  /s): min=65024, max=70648, per=25.01%, avg=67931.39, stdev=718.44
    lat (usec) : 20=0.01%, 250=0.01%, 500=1.25%, 750=5.87%, 1000=9.30%
    lat (msec) : 2=43.54%, 4=37.90%, 10=2.13%, 20=0.01%
  cpu          : usr=16.75%, sys=101.01%, ctx=17967756, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=71.5%, 32=28.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.9%, 8=4.3%, 16=1.7%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=4074113/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq-write: (groupid=2, jobs=4): err= 0: pid=48697: Wed Dec  2 13:01:17 2015
  write: io=32768MB, bw=719049KB/s, iops=179762, runt= 46665msec
    slat (usec): min=0, max=11428, avg=11.03, stdev=49.70
    clat (usec): min=1, max=20966, avg=592.92, stdev=805.28
     lat (usec): min=19, max=20970, avg=603.95, stdev=804.76
    clat percentiles (usec):
     |  1.00th=[   31],  5.00th=[   65], 10.00th=[   93], 20.00th=[  149],
     | 30.00th=[  207], 40.00th=[  278], 50.00th=[  362], 60.00th=[  466],
     | 70.00th=[  604], 80.00th=[  796], 90.00th=[ 1176], 95.00th=[ 1768],
     | 99.00th=[ 4640], 99.50th=[ 5408], 99.90th=[ 6944], 99.95th=[ 7904],
     | 99.99th=[10560]
    bw (KB  /s): min=157440, max=209376, per=25.02%, avg=179918.88, stdev=10440.64
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.15%, 20=0.38%, 50=2.18%
    lat (usec) : 100=8.46%, 250=25.16%, 500=26.30%, 750=15.24%, 1000=8.49%
    lat (msec) : 2=9.44%, 4=2.67%, 10=1.51%, 20=0.02%, 50=0.01%
  cpu          : usr=35.97%, sys=639.05%, ctx=13361261, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.1%, 8=8.5%, 16=78.6%, 32=11.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.7%, 8=2.3%, 16=2.3%, 32=1.7%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=8388608/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand-write: (groupid=3, jobs=4): err= 0: pid=48967: Wed Dec  2 13:01:17 2015
  write: io=4936.1MB, bw=84234KB/s, iops=21058, runt= 60017msec
    slat (usec): min=0, max=1002, avg= 4.55, stdev= 7.53
    clat (usec): min=1, max=81442, avg=6054.34, stdev=7574.50
     lat (usec): min=19, max=81444, avg=6058.89, stdev=7574.38
    clat percentiles (usec):
     |  1.00th=[   31],  5.00th=[   90], 10.00th=[  442], 20.00th=[  820],
     | 30.00th=[ 1240], 40.00th=[ 1832], 50.00th=[ 2512], 60.00th=[ 3856],
     | 70.00th=[ 6560], 80.00th=[11072], 90.00th=[17792], 95.00th=[22144],
     | 99.00th=[32384], 99.50th=[36096], 99.90th=[44288], 99.95th=[47872],
     | 99.99th=[55552]
    bw (KB  /s): min=18176, max=24336, per=25.03%, avg=21082.86, stdev=919.26
    lat (usec) : 2=0.01%, 4=0.03%, 10=0.02%, 20=0.05%, 50=2.86%
    lat (usec) : 100=2.30%, 250=2.47%, 500=3.49%, 750=6.89%, 1000=6.58%
    lat (msec) : 2=17.82%, 4=18.30%, 10=17.26%, 20=14.88%, 50=7.01%
    lat (msec) : 100=0.03%
  cpu          : usr=5.51%, sys=41.60%, ctx=4626054, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=71.6%, 32=28.3%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.9%, 8=4.4%, 16=1.7%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1263862/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=32768MB, aggrb=729079KB/s, minb=729079KB/s, maxb=729079KB/s, mint=46023msec, maxt=46023msec

Run status group 1 (all jobs):
   READ: io=15915MB, aggrb=271593KB/s, minb=271593KB/s, maxb=271593KB/s, mint=60003msec, maxt=60003msec

Run status group 2 (all jobs):
  WRITE: io=32768MB, aggrb=719049KB/s, minb=719049KB/s, maxb=719049KB/s, mint=46665msec, maxt=46665msec

Run status group 3 (all jobs):
  WRITE: io=4936.1MB, aggrb=84233KB/s, minb=84233KB/s, maxb=84233KB/s, mint=60017msec, maxt=60017msec