Intel Optane DC P4800X 375GB

P4800X.log

Intel Optane DC P4800X 375GB
user@ubuntu:/tmp$ ~/fio/fio --loops=5 --size=1g --runtime=180 --stonewall --direct=1 --group_reporting \
>     --name=SeqQ32T1read --bs=128k --iodepth=32 --rw=read \
>     --name=SeqQ32T1write --bs=128k --iodepth=32 --rw=write \
>     --name=4kQ8T8read --bs=4k --iodepth=8 --numjobs=8 --rw=randread \
>     --name=4kQ8T8write --bs=4k --iodepth=8 --numjobs=8 --rw=randwrite \
>     --name=4kQ32T1read --bs=4k --iodepth=32 --rw=randread \
>     --name=4kQ32T1write --bs=4k --iodepth=32 --rw=randwrite \
>     --name=4kQ1T1read --bs=4k --iodepth=1 --rw=randread \
>     --name=4kQ1T1write --bs=4k --iodepth=1 --rw=randwrite
SeqQ32T1read: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=32
SeqQ32T1write: (g=1): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=32
4kQ8T8read: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=8
4kQ8T8write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=8
4kQ32T1read: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
4kQ32T1write: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=32
4kQ1T1read: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
4kQ1T1write: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.14
Starting 22 processes
SeqQ32T1read: Laying out IO file (1 file / 1024MiB)
SeqQ32T1write: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8read: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ8T8write: Laying out IO file (1 file / 1024MiB)
4kQ32T1read: Laying out IO file (1 file / 1024MiB)
4kQ32T1write: Laying out IO file (1 file / 1024MiB)
4kQ1T1read: Laying out IO file (1 file / 1024MiB)
4kQ1T1write: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [_(21),w(1)][100.0%][w=266MiB/s][w=68.1k IOPS][eta 00m:00s]
SeqQ32T1read: (groupid=0, jobs=1): err= 0: pid=17770: Tue Jun 18 11:38:54 2019
  read: IOPS=11.8k, BW=1475MiB/s (1547MB/s)(5120MiB/3471msec)
    clat (usec): min=77, max=240, avg=83.32, stdev= 6.55
     lat (usec): min=77, max=241, avg=83.44, stdev= 6.56
    clat percentiles (usec):
     |  1.00th=[   79],  5.00th=[   79], 10.00th=[   80], 20.00th=[   81],
     | 30.00th=[   82], 40.00th=[   82], 50.00th=[   82], 60.00th=[   82],
     | 70.00th=[   82], 80.00th=[   83], 90.00th=[   95], 95.00th=[   96],
     | 99.00th=[  113], 99.50th=[  121], 99.90th=[  130], 99.95th=[  133],
     | 99.99th=[  147]
   bw (  MiB/s): min=  985, max= 1474, per=72.41%, avg=1068.06, stdev=199.00, samples=6
   iops        : min= 7885, max=11794, avg=8544.17, stdev=1592.12, samples=6
  lat (usec)   : 100=98.14%, 250=1.86%
  cpu          : usr=2.54%, sys=26.02%, ctx=40966, majf=0, minf=42
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
SeqQ32T1write: (groupid=1, jobs=1): err= 0: pid=17795: Tue Jun 18 11:38:54 2019
  write: IOPS=9810, BW=1226MiB/s (1286MB/s)(5120MiB/4175msec); 0 zone resets
    clat (usec): min=78, max=283, avg=96.24, stdev=25.66
     lat (usec): min=80, max=287, avg=100.53, stdev=25.63
    clat percentiles (usec):
     |  1.00th=[   80],  5.00th=[   80], 10.00th=[   81], 20.00th=[   84],
     | 30.00th=[   85], 40.00th=[   86], 50.00th=[   86], 60.00th=[   87],
     | 70.00th=[   92], 80.00th=[   96], 90.00th=[  133], 95.00th=[  167],
     | 99.00th=[  190], 99.50th=[  198], 99.90th=[  229], 99.95th=[  241],
     | 99.99th=[  269]
   bw (  MiB/s): min= 1033, max= 1195, per=94.38%, avg=1157.47, stdev=65.29, samples=8
   iops        : min= 8267, max= 9567, avg=9259.00, stdev=522.29, samples=8
  lat (usec)   : 100=83.22%, 250=16.76%, 500=0.03%
  cpu          : usr=7.91%, sys=20.36%, ctx=40973, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ8T8read: (groupid=2, jobs=8): err= 0: pid=17822: Tue Jun 18 11:38:54 2019
  read: IOPS=490k, BW=1913MiB/s (2006MB/s)(40.0GiB/21414msec)
    clat (usec): min=10, max=450, avg=15.51, stdev= 3.38
     lat (usec): min=10, max=450, avg=15.57, stdev= 3.38
    clat percentiles (nsec):
     |  1.00th=[13376],  5.00th=[13632], 10.00th=[13760], 20.00th=[14016],
     | 30.00th=[14144], 40.00th=[14400], 50.00th=[14656], 60.00th=[14912],
     | 70.00th=[15296], 80.00th=[15936], 90.00th=[17024], 95.00th=[21632],
     | 99.00th=[28288], 99.50th=[34048], 99.90th=[44800], 99.95th=[49920],
     | 99.99th=[57600]
   bw (  MiB/s): min= 1465, max= 1891, per=82.62%, avg=1580.30, stdev= 7.42, samples=336
   iops        : min=375278, max=484221, avg=404554.48, stdev=1899.67, samples=336
  lat (usec)   : 20=94.52%, 50=5.43%, 100=0.05%, 250=0.01%, 500=0.01%
  cpu          : usr=8.28%, sys=31.36%, ctx=10486339, majf=0, minf=1582
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10485760,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ8T8write: (groupid=3, jobs=8): err= 0: pid=17956: Tue Jun 18 11:38:54 2019
  write: IOPS=334k, BW=1305MiB/s (1369MB/s)(40.0GiB/31376msec); 0 zone resets
    clat (usec): min=13, max=2119, avg=22.93, stdev=11.48
     lat (usec): min=13, max=2120, avg=23.02, stdev=11.48
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   18], 10.00th=[   18], 20.00th=[   19],
     | 30.00th=[   19], 40.00th=[   19], 50.00th=[   20], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   24], 90.00th=[   27], 95.00th=[   58],
     | 99.00th=[   70], 99.50th=[   74], 99.90th=[   98], 99.95th=[  105],
     | 99.99th=[  123]
   bw (  KiB/s): min=700864, max=1055991, per=70.63%, avg=944146.55, stdev=13954.79, samples=491
   iops        : min=175214, max=263995, avg=236033.60, stdev=3488.69, samples=491
  lat (usec)   : 20=58.24%, 50=35.97%, 100=5.71%, 250=0.08%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=6.53%, sys=37.23%, ctx=10486313, majf=0, minf=1290
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10485760,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ32T1read: (groupid=4, jobs=1): err= 0: pid=18150: Tue Jun 18 11:38:54 2019
  read: IOPS=77.1k, BW=301MiB/s (316MB/s)(5120MiB/16992msec)
    clat (usec): min=10, max=101, avg=12.36, stdev= 1.52
     lat (usec): min=10, max=101, avg=12.40, stdev= 1.53
    clat percentiles (nsec):
     |  1.00th=[11712],  5.00th=[11840], 10.00th=[11840], 20.00th=[11968],
     | 30.00th=[11968], 40.00th=[12096], 50.00th=[12224], 60.00th=[12224],
     | 70.00th=[12224], 80.00th=[12352], 90.00th=[12608], 95.00th=[13120],
     | 99.00th=[17536], 99.50th=[20608], 99.90th=[37120], 99.95th=[39680],
     | 99.99th=[43776]
   bw (  KiB/s): min=259248, max=295879, per=89.14%, avg=275037.45, stdev=6432.89, samples=33
   iops        : min=64812, max=73969, avg=68758.91, stdev=1608.10, samples=33
  lat (usec)   : 20=99.20%, 50=0.80%, 100=0.01%, 250=0.01%
  cpu          : usr=9.38%, sys=28.99%, ctx=1310735, majf=0, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ32T1write: (groupid=5, jobs=1): err= 0: pid=18253: Tue Jun 18 11:38:54 2019
  write: IOPS=64.6k, BW=252MiB/s (265MB/s)(5120MiB/20293msec); 0 zone resets
    clat (usec): min=11, max=181, avg=14.77, stdev= 4.20
     lat (usec): min=11, max=181, avg=14.85, stdev= 4.21
    clat percentiles (nsec):
     |  1.00th=[12992],  5.00th=[13248], 10.00th=[13248], 20.00th=[13376],
     | 30.00th=[13504], 40.00th=[13504], 50.00th=[13632], 60.00th=[13760],
     | 70.00th=[14144], 80.00th=[16064], 90.00th=[17024], 95.00th=[17536],
     | 99.00th=[37120], 99.50th=[51968], 99.90th=[57600], 99.95th=[60672],
     | 99.99th=[63744]
   bw (  KiB/s): min=179229, max=236163, per=85.59%, avg=221126.35, stdev=18995.58, samples=40
   iops        : min=44807, max=59040, avg=55281.13, stdev=4749.03, samples=40
  lat (usec)   : 20=97.88%, 50=1.37%, 100=0.75%, 250=0.01%
  cpu          : usr=8.07%, sys=34.25%, ctx=1310737, majf=0, minf=87
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ1T1read: (groupid=6, jobs=1): err= 0: pid=18374: Tue Jun 18 11:38:54 2019
  read: IOPS=79.4k, BW=310MiB/s (325MB/s)(5120MiB/16518msec)
    clat (usec): min=10, max=139, avg=12.01, stdev= 1.61
     lat (usec): min=10, max=139, avg=12.05, stdev= 1.62
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11200], 10.00th=[11200], 20.00th=[11328],
     | 30.00th=[11456], 40.00th=[11584], 50.00th=[11840], 60.00th=[11968],
     | 70.00th=[12096], 80.00th=[12224], 90.00th=[12480], 95.00th=[12992],
     | 99.00th=[18816], 99.50th=[20352], 99.90th=[36608], 99.95th=[39680],
     | 99.99th=[43776]
   bw (  KiB/s): min=265859, max=325448, per=90.47%, avg=287141.28, stdev=12567.70, samples=32
   iops        : min=66464, max=81362, avg=71784.97, stdev=3142.02, samples=32
  lat (usec)   : 20=99.29%, 50=0.71%, 100=0.01%, 250=0.01%
  cpu          : usr=12.48%, sys=25.69%, ctx=1310737, majf=0, minf=177
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQ1T1write: (groupid=7, jobs=1): err= 0: pid=18471: Tue Jun 18 11:38:54 2019
  write: IOPS=65.1k, BW=254MiB/s (267MB/s)(5120MiB/20136msec); 0 zone resets
    clat (usec): min=11, max=264, avg=14.69, stdev= 4.07
     lat (usec): min=11, max=264, avg=14.75, stdev= 4.07
    clat percentiles (nsec):
     |  1.00th=[13120],  5.00th=[13248], 10.00th=[13376], 20.00th=[13504],
     | 30.00th=[13504], 40.00th=[13632], 50.00th=[13632], 60.00th=[13760],
     | 70.00th=[14016], 80.00th=[15808], 90.00th=[16512], 95.00th=[17280],
     | 99.00th=[35072], 99.50th=[51968], 99.90th=[58624], 99.95th=[60672],
     | 99.99th=[63744]
   bw (  KiB/s): min=213680, max=276168, per=99.95%, avg=260255.17, stdev=20223.49, samples=40
   iops        : min=53420, max=69044, avg=65063.85, stdev=5055.84, samples=40
  lat (usec)   : 20=98.21%, 50=1.05%, 100=0.74%, 250=0.01%, 500=0.01%
  cpu          : usr=7.40%, sys=34.62%, ctx=1310738, majf=0, minf=152
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1475MiB/s (1547MB/s), 1475MiB/s-1475MiB/s (1547MB/s-1547MB/s), io=5120MiB (5369MB), run=3471-3471msec

Run status group 1 (all jobs):
  WRITE: bw=1226MiB/s (1286MB/s), 1226MiB/s-1226MiB/s (1286MB/s-1286MB/s), io=5120MiB (5369MB), run=4175-4175msec

Run status group 2 (all jobs):
   READ: bw=1913MiB/s (2006MB/s), 1913MiB/s-1913MiB/s (2006MB/s-2006MB/s), io=40.0GiB (42.9GB), run=21414-21414msec

Run status group 3 (all jobs):
  WRITE: bw=1305MiB/s (1369MB/s), 1305MiB/s-1305MiB/s (1369MB/s-1369MB/s), io=40.0GiB (42.9GB), run=31376-31376msec

Run status group 4 (all jobs):
   READ: bw=301MiB/s (316MB/s), 301MiB/s-301MiB/s (316MB/s-316MB/s), io=5120MiB (5369MB), run=16992-16992msec

Run status group 5 (all jobs):
  WRITE: bw=252MiB/s (265MB/s), 252MiB/s-252MiB/s (265MB/s-265MB/s), io=5120MiB (5369MB), run=20293-20293msec

Run status group 6 (all jobs):
   READ: bw=310MiB/s (325MB/s), 310MiB/s-310MiB/s (325MB/s-325MB/s), io=5120MiB (5369MB), run=16518-16518msec

Run status group 7 (all jobs):
  WRITE: bw=254MiB/s (267MB/s), 254MiB/s-254MiB/s (267MB/s-267MB/s), io=5120MiB (5369MB), run=20136-20136msec

Disk stats (read/write):
  nvme6n1: ios=13148160/13145136, merge=0/17387, ticks=133784/176844, in_queue=109248, util=36.06%