Intel Optane 900P 480GB test

900p.log

900p 480GB
SeqQ32T1read: (groupid=0, jobs=1): err= 0: pid=29758: Tue Jan  8 06:12:55 2019
  read: IOPS=15.1k, BW=1884MiB/s (1976MB/s)(5120MiB/2717msec)
    clat (usec): min=62, max=348, avg=65.73, stdev= 5.52
     lat (usec): min=62, max=348, avg=65.81, stdev= 5.52
    clat percentiles (usec):
     |  1.00th=[   63],  5.00th=[   63], 10.00th=[   63], 20.00th=[   64],
     | 30.00th=[   64], 40.00th=[   64], 50.00th=[   65], 60.00th=[   65],
     | 70.00th=[   65], 80.00th=[   65], 90.00th=[   75], 95.00th=[   76],
     | 99.00th=[   93], 99.50th=[   95], 99.90th=[  105], 99.95th=[  109],
     | 99.99th=[  124]
   bw (  MiB/s): min= 1499, max= 1882, per=83.68%, avg=1576.91, stdev=170.87, samples=5
   iops        : min=11995, max=15060, avg=12614.80, stdev=1366.94, samples=5
  lat (usec)   : 100=99.85%, 250=0.15%, 500=0.01%
  cpu          : usr=5.60%, sys=10.16%, ctx=40964, majf=0, minf=43
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
SeqQ32T1write: (groupid=1, jobs=1): err= 0: pid=29759: Tue Jan  8 06:12:55 2019
  write: IOPS=13.3k, BW=1659MiB/s (1739MB/s)(5120MiB/3087msec); 0 zone resets
    clat (usec): min=62, max=271, avg=73.99, stdev=13.58
     lat (usec): min=63, max=272, avg=74.77, stdev=13.58
    clat percentiles (usec):
     |  1.00th=[   66],  5.00th=[   66], 10.00th=[   66], 20.00th=[   67],
     | 30.00th=[   67], 40.00th=[   69], 50.00th=[   69], 60.00th=[   70],
     | 70.00th=[   70], 80.00th=[   80], 90.00th=[  101], 95.00th=[  103],
     | 99.00th=[  116], 99.50th=[  126], 99.90th=[  137], 99.95th=[  141],
     | 99.99th=[  159]
   bw (  MiB/s): min= 1329, max= 1658, per=84.31%, avg=1398.39, stdev=145.46, samples=5
   iops        : min=10635, max=13268, avg=11186.80, stdev=1163.58, samples=5
  lat (usec)   : 100=89.10%, 250=10.90%, 500=0.01%
  cpu          : usr=3.99%, sys=10.37%, ctx=40966, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ8T8read: (groupid=2, jobs=8): err= 0: pid=29760: Tue Jan  8 06:12:55 2019
  read: IOPS=538k, BW=2100MiB/s (2202MB/s)(40.0GiB/19502msec)
    clat (usec): min=10, max=1287, avg=14.19, stdev= 2.99
     lat (usec): min=10, max=1287, avg=14.26, stdev= 2.99
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[12352], 10.00th=[12608], 20.00th=[12864],
     | 30.00th=[13120], 40.00th=[13376], 50.00th=[13504], 60.00th=[13760],
     | 70.00th=[14016], 80.00th=[14528], 90.00th=[15296], 95.00th=[17280],
     | 99.00th=[25216], 99.50th=[27520], 99.90th=[43264], 99.95th=[44288],
     | 99.99th=[51968]
   bw (  KiB/s): min=220985, max=274553, per=11.17%, avg=240296.73, stdev=8364.10, samples=304
   iops        : min=55246, max=68638, avg=60073.82, stdev=2091.01, samples=304
  lat (usec)   : 20=95.37%, 50=4.61%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=0.91%, sys=32.78%, ctx=10486405, majf=0, minf=1707
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10485760,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ8T8write: (groupid=3, jobs=8): err= 0: pid=29768: Tue Jan  8 06:12:55 2019
  write: IOPS=435k, BW=1700MiB/s (1782MB/s)(40.0GiB/24097msec); 0 zone resets
    clat (usec): min=11, max=1260, avg=17.43, stdev= 8.29
     lat (usec): min=11, max=1260, avg=17.54, stdev= 8.29
    clat percentiles (nsec):
     |  1.00th=[13376],  5.00th=[13888], 10.00th=[14144], 20.00th=[14400],
     | 30.00th=[14784], 40.00th=[15040], 50.00th=[15424], 60.00th=[15680],
     | 70.00th=[16064], 80.00th=[16512], 90.00th=[18048], 95.00th=[42752],
     | 99.00th=[53504], 99.50th=[55040], 99.90th=[71168], 99.95th=[79360],
     | 99.99th=[83456]
   bw (  KiB/s): min=173203, max=226664, per=11.03%, avg=191996.90, stdev=5403.50, samples=377
   iops        : min=43300, max=56666, avg=47998.86, stdev=1350.88, samples=377
  lat (usec)   : 20=93.59%, 50=2.71%, 100=3.70%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=0.96%, sys=31.63%, ctx=10486693, majf=0, minf=1284
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10485760,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ32T1read: (groupid=4, jobs=1): err= 0: pid=29776: Tue Jan  8 06:12:55 2019
  read: IOPS=81.9k, BW=320MiB/s (336MB/s)(5120MiB/16000msec)
    clat (usec): min=10, max=317, avg=11.69, stdev= 1.46
     lat (usec): min=10, max=317, avg=11.76, stdev= 1.46
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11328], 20.00th=[11328],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11712], 90.00th=[11840], 95.00th=[12224],
     | 99.00th=[14528], 99.50th=[15424], 99.90th=[37120], 99.95th=[39168],
     | 99.99th=[41216]
   bw (  KiB/s): min=215572, max=325326, per=67.90%, avg=222490.68, stdev=19179.78, samples=31
   iops        : min=53893, max=81331, avg=55622.29, stdev=4794.92, samples=31
  lat (usec)   : 20=99.68%, 50=0.32%, 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=0.08%, sys=36.00%, ctx=1310726, majf=0, minf=172
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ32T1write: (groupid=5, jobs=1): err= 0: pid=29779: Tue Jan  8 06:12:55 2019
  write: IOPS=65.3k, BW=255MiB/s (267MB/s)(5120MiB/20078msec); 0 zone resets
    clat (usec): min=11, max=870, avg=14.70, stdev= 4.62
     lat (usec): min=11, max=870, avg=14.79, stdev= 4.63
    clat percentiles (nsec):
     |  1.00th=[12352],  5.00th=[12480], 10.00th=[12480], 20.00th=[12480],
     | 30.00th=[12608], 40.00th=[12608], 50.00th=[12608], 60.00th=[12736],
     | 70.00th=[12864], 80.00th=[19328], 90.00th=[21376], 95.00th=[22400],
     | 99.00th=[32640], 99.50th=[41728], 99.90th=[44800], 99.95th=[45824],
     | 99.99th=[51968]
   bw (  KiB/s): min=160247, max=280060, per=94.12%, avg=245768.15, stdev=50040.49, samples=40
   iops        : min=40061, max=70015, avg=61441.75, stdev=12510.14, samples=40
  lat (usec)   : 20=82.15%, 50=17.83%, 100=0.02%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  cpu          : usr=1.93%, sys=41.55%, ctx=1310740, majf=0, minf=111
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ1T1read: (groupid=6, jobs=1): err= 0: pid=88000: Thu Feb 21 08:26:56 2019
  read: IOPS=82.6k, BW=323MiB/s (338MB/s)(5120MiB/15873msec)
    clat (usec): min=10, max=644, avg=11.60, stdev= 1.62
     lat (usec): min=10, max=644, avg=11.66, stdev= 1.62
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11200], 10.00th=[11200], 20.00th=[11328],
     | 30.00th=[11328], 40.00th=[11328], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11840], 95.00th=[12352],
     | 99.00th=[14912], 99.50th=[18304], 99.90th=[37120], 99.95th=[39168],
     | 99.99th=[41216]
   bw (  KiB/s): min=266870, max=326075, per=82.77%, avg=273374.94, stdev=10059.08, samples=31
   iops        : min=66717, max=81520, avg=68343.29, stdev=2515.10, samples=31
  lat (usec)   : 20=99.64%, 50=0.35%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%
  cpu          : usr=0.09%, sys=35.74%, ctx=1310730, majf=0, minf=101
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQ1T1write: (groupid=7, jobs=1): err= 0: pid=88001: Thu Feb 21 08:26:56 2019
  write: IOPS=65.0k, BW=258MiB/s (270MB/s)(5120MiB/19863msec); 0 zone resets
    clat (usec): min=11, max=795, avg=14.56, stdev= 4.72
     lat (usec): min=11, max=795, avg=14.63, stdev= 4.73
    clat percentiles (nsec):
     |  1.00th=[12096],  5.00th=[12224], 10.00th=[12224], 20.00th=[12352],
     | 30.00th=[12352], 40.00th=[12352], 50.00th=[12480], 60.00th=[12480],
     | 70.00th=[12736], 80.00th=[19328], 90.00th=[21376], 95.00th=[22400],
     | 99.00th=[35072], 99.50th=[41728], 99.90th=[44288], 99.95th=[45824],
     | 99.99th=[52480]
   bw (  KiB/s): min=169088, max=302912, per=99.74%, avg=263260.08, stdev=56584.71, samples=39
   iops        : min=42272, max=75728, avg=65815.00, stdev=14146.21, samples=39
  lat (usec)   : 20=82.75%, 50=17.23%, 100=0.02%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  cpu          : usr=1.29%, sys=41.88%, ctx=1310736, majf=0, minf=98
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1884MiB/s (1976MB/s), 1884MiB/s-1884MiB/s (1976MB/s-1976MB/s), io=5120MiB (5369MB), run=2717-2717msec

Run status group 1 (all jobs):
  WRITE: bw=1659MiB/s (1739MB/s), 1659MiB/s-1659MiB/s (1739MB/s-1739MB/s), io=5120MiB (5369MB), run=3087-3087msec

Run status group 2 (all jobs):
   READ: bw=2100MiB/s (2202MB/s), 2100MiB/s-2100MiB/s (2202MB/s-2202MB/s), io=40.0GiB (42.9GB), run=19502-19502msec

Run status group 3 (all jobs):
  WRITE: bw=1700MiB/s (1782MB/s), 1700MiB/s-1700MiB/s (1782MB/s-1782MB/s), io=40.0GiB (42.9GB), run=24097-24097msec

Run status group 4 (all jobs):
   READ: bw=320MiB/s (336MB/s), 320MiB/s-320MiB/s (336MB/s-336MB/s), io=5120MiB (5369MB), run=16000-16000msec

Run status group 5 (all jobs):
  WRITE: bw=255MiB/s (267MB/s), 255MiB/s-255MiB/s (267MB/s-267MB/s), io=5120MiB (5369MB), run=20078-20078msec

Run status group 6 (all jobs):
   READ: bw=323MiB/s (338MB/s), 323MiB/s-323MiB/s (338MB/s-338MB/s), io=5120MiB (5369MB), run=15873-15873msec

Run status group 7 (all jobs):
  WRITE: bw=258MiB/s (270MB/s), 258MiB/s-258MiB/s (270MB/s-270MB/s), io=5120MiB (5369MB), run=19863-19863msec

Disk stats (read/write):
  nvme3n1: ios=13148160/13139067, merge=0/17927, ticks=129432/158340, in_queue=111380, util=34.96%