4x Micron Crucial P1 1TB mdadm raid0 test

4xP1-1tb.log

4x P1 1TB
SeqQ32T1read: (groupid=0, jobs=1): err= 0: pid=29929: Tue Jan  8 06:20:35 2019
  read: IOPS=10.5k, BW=1313MiB/s (1377MB/s)(5120MiB/3900msec)
    clat (usec): min=58, max=2411, avg=94.64, stdev=52.07
     lat (usec): min=58, max=2411, avg=94.70, stdev=52.07
    clat percentiles (usec):
     |  1.00th=[   60],  5.00th=[   60], 10.00th=[   60], 20.00th=[   60],
     | 30.00th=[   61], 40.00th=[   63], 50.00th=[   77], 60.00th=[   93],
     | 70.00th=[  110], 80.00th=[  120], 90.00th=[  139], 95.00th=[  182],
     | 99.00th=[  310], 99.50th=[  343], 99.90th=[  351], 99.95th=[  351],
     | 99.99th=[  947]
   bw (  MiB/s): min= 1016, max= 1315, per=81.07%, avg=1064.28, stdev=110.95, samples=7
   iops        : min= 8135, max=10525, avg=8513.71, stdev=887.46, samples=7
  lat (usec)   : 100=63.91%, 250=33.88%, 500=2.19%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=3.85%, sys=8.00%, ctx=40964, majf=0, minf=44
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
SeqQ32T1write: (groupid=1, jobs=1): err= 0: pid=29930: Tue Jan  8 06:20:35 2019
  write: IOPS=12.8k, BW=1599MiB/s (1677MB/s)(5120MiB/3202msec); 0 zone resets
    clat (usec): min=60, max=13955, avg=76.84, stdev=231.68
     lat (usec): min=61, max=13956, avg=77.61, stdev=231.68
    clat percentiles (usec):
     |  1.00th=[   62],  5.00th=[   62], 10.00th=[   62], 20.00th=[   62],
     | 30.00th=[   63], 40.00th=[   64], 50.00th=[   66], 60.00th=[   70],
     | 70.00th=[   77], 80.00th=[   87], 90.00th=[   92], 95.00th=[   94],
     | 99.00th=[   96], 99.50th=[   97], 99.90th=[  101], 99.95th=[ 2704],
     | 99.99th=[12649]
   bw (  MiB/s): min= 1467, max= 1560, per=94.53%, avg=1511.59, stdev=38.61, samples=6
   iops        : min=11737, max=12481, avg=12092.17, stdev=308.93, samples=6
  lat (usec)   : 100=99.90%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%
  cpu          : usr=4.62%, sys=10.18%, ctx=40964, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ8T8read: (groupid=2, jobs=8): err= 0: pid=29931: Tue Jan  8 06:20:35 2019
  read: IOPS=91.6k, BW=358MiB/s (375MB/s)(40.0GiB/114415msec)
    clat (usec): min=13, max=3252, avg=79.80, stdev=48.71
     lat (usec): min=13, max=3252, avg=79.88, stdev=48.71
    clat percentiles (usec):
     |  1.00th=[   57],  5.00th=[   57], 10.00th=[   58], 20.00th=[   58],
     | 30.00th=[   58], 40.00th=[   59], 50.00th=[   59], 60.00th=[   61],
     | 70.00th=[   72], 80.00th=[   90], 90.00th=[  135], 95.00th=[  198],
     | 99.00th=[  277], 99.50th=[  318], 99.90th=[  408], 99.95th=[  441],
     | 99.99th=[  594]
   bw (  KiB/s): min=40428, max=53905, per=12.92%, avg=47377.22, stdev=3571.36, samples=1684
   iops        : min=10107, max=13476, avg=11843.93, stdev=892.83, samples=1684
  lat (usec)   : 20=0.01%, 50=0.02%, 100=84.78%, 250=13.71%, 500=1.48%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=0.74%, sys=8.00%, ctx=10486090, majf=0, minf=1571
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10485760,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ8T8write: (groupid=3, jobs=8): err= 0: pid=29943: Tue Jan  8 06:20:35 2019
  write: IOPS=365k, BW=1427MiB/s (1497MB/s)(40.0GiB/28699msec); 0 zone resets
    clat (usec): min=11, max=13603, avg=20.94, stdev=102.22
     lat (usec): min=11, max=13603, avg=21.04, stdev=102.22
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   14], 10.00th=[   14], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   16], 50.00th=[   17], 60.00th=[   18],
     | 70.00th=[   24], 80.00th=[   26], 90.00th=[   32], 95.00th=[   37],
     | 99.00th=[   43], 99.50th=[   47], 99.90th=[   53], 99.95th=[   61],
     | 99.99th=[ 1844]
   bw (  KiB/s): min=103394, max=201256, per=11.38%, avg=166385.50, stdev=27456.72, samples=450
   iops        : min=25848, max=50314, avg=41596.00, stdev=6864.18, samples=450
  lat (usec)   : 20=64.99%, 50=34.81%, 100=0.17%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=2.88%, sys=39.11%, ctx=10487202, majf=0, minf=2047
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10485760,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
4kQ32T1read: (groupid=4, jobs=1): err= 0: pid=29954: Tue Jan  8 06:20:35 2019
  read: IOPS=17.5k, BW=68.3MiB/s (71.6MB/s)(5120MiB/74937msec)
    clat (usec): min=14, max=25735, avg=56.51, stdev=25.92
     lat (usec): min=14, max=25735, avg=56.58, stdev=25.92
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   56], 10.00th=[   56], 20.00th=[   57],
     | 30.00th=[   57], 40.00th=[   57], 50.00th=[   57], 60.00th=[   57],
     | 70.00th=[   57], 80.00th=[   57], 90.00th=[   58], 95.00th=[   58],
     | 99.00th=[   60], 99.50th=[   61], 99.90th=[   84], 99.95th=[   85],
     | 99.99th=[  383]
   bw (  KiB/s): min=56798, max=67510, per=81.91%, avg=57310.08, stdev=869.79, samples=149
   iops        : min=14199, max=16877, avg=14327.13, stdev=217.45, samples=149
  lat (usec)   : 20=0.01%, 50=0.15%, 100=99.83%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 50=0.01%
  cpu          : usr=0.04%, sys=10.31%, ctx=1310753, majf=0, minf=183
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ32T1write: (groupid=5, jobs=1): err= 0: pid=29955: Tue Jan  8 06:20:35 2019
  write: IOPS=63.7k, BW=249MiB/s (261MB/s)(5120MiB/20566msec); 0 zone resets
    clat (usec): min=11, max=36434, avg=15.05, stdev=51.72
     lat (usec): min=11, max=36434, avg=15.14, stdev=51.72
    clat percentiles (nsec):
     |  1.00th=[12352],  5.00th=[12480], 10.00th=[12608], 20.00th=[12736],
     | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60.00th=[13120],
     | 70.00th=[13248], 80.00th=[19328], 90.00th=[21376], 95.00th=[22400],
     | 99.00th=[26240], 99.50th=[27008], 99.90th=[31104], 99.95th=[35584],
     | 99.99th=[72192]
   bw (  KiB/s): min=122739, max=215183, per=72.88%, avg=185793.20, stdev=33932.50, samples=40
   iops        : min=30684, max=53795, avg=46447.95, stdev=8483.13, samples=40
  lat (usec)   : 20=82.53%, 50=17.45%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.56%, sys=43.34%, ctx=1310744, majf=0, minf=192
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
4kQ1T1read: (groupid=6, jobs=1): err= 0: pid=29956: Tue Jan  8 06:20:35 2019
  read: IOPS=17.3k, BW=67.6MiB/s (70.9MB/s)(5120MiB/75766msec)
    clat (usec): min=13, max=27968, avg=57.14, stdev=40.53
     lat (usec): min=13, max=27968, avg=57.21, stdev=40.53
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   57], 10.00th=[   57], 20.00th=[   57],
     | 30.00th=[   57], 40.00th=[   57], 50.00th=[   58], 60.00th=[   58],
     | 70.00th=[   58], 80.00th=[   58], 90.00th=[   58], 95.00th=[   59],
     | 99.00th=[   61], 99.50th=[   61], 99.90th=[   85], 99.95th=[   86],
     | 99.99th=[  404]
   bw (  KiB/s): min=55521, max=61629, per=80.89%, avg=55977.38, stdev=524.97, samples=151
   iops        : min=13880, max=15407, avg=13994.00, stdev=131.23, samples=151
  lat (usec)   : 20=0.01%, 50=0.14%, 100=99.84%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 50=0.01%
  cpu          : usr=0.03%, sys=10.29%, ctx=1310748, majf=0, minf=208
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQ1T1write: (groupid=7, jobs=1): err= 0: pid=29957: Tue Jan  8 06:20:35 2019
  write: IOPS=63.2k, BW=247MiB/s (259MB/s)(5120MiB/20741msec); 0 zone resets
    clat (usec): min=11, max=46302, avg=15.20, stdev=69.56
     lat (usec): min=11, max=46302, avg=15.28, stdev=69.56
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   13], 10.00th=[   13], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   13], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   14], 80.00th=[   20], 90.00th=[   22], 95.00th=[   23],
     | 99.00th=[   28], 99.50th=[   28], 99.90th=[   33], 99.95th=[   38],
     | 99.99th=[  103]
   bw (  KiB/s): min=135896, max=297744, per=99.83%, avg=252336.95, stdev=52759.14, samples=41
   iops        : min=33974, max=74436, avg=63084.20, stdev=13189.80, samples=41
  lat (usec)   : 20=80.17%, 50=19.81%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.77%, sys=42.80%, ctx=1310742, majf=0, minf=229
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1313MiB/s (1377MB/s), 1313MiB/s-1313MiB/s (1377MB/s-1377MB/s), io=5120MiB (5369MB), run=3900-3900msec

Run status group 1 (all jobs):
  WRITE: bw=1599MiB/s (1677MB/s), 1599MiB/s-1599MiB/s (1677MB/s-1677MB/s), io=5120MiB (5369MB), run=3202-3202msec

Run status group 2 (all jobs):
   READ: bw=358MiB/s (375MB/s), 358MiB/s-358MiB/s (375MB/s-375MB/s), io=40.0GiB (42.9GB), run=114415-114415msec

Run status group 3 (all jobs):
  WRITE: bw=1427MiB/s (1497MB/s), 1427MiB/s-1427MiB/s (1497MB/s-1497MB/s), io=40.0GiB (42.9GB), run=28699-28699msec

Run status group 4 (all jobs):
   READ: bw=68.3MiB/s (71.6MB/s), 68.3MiB/s-68.3MiB/s (71.6MB/s-71.6MB/s), io=5120MiB (5369MB), run=74937-74937msec

Run status group 5 (all jobs):
  WRITE: bw=249MiB/s (261MB/s), 249MiB/s-249MiB/s (261MB/s-261MB/s), io=5120MiB (5369MB), run=20566-20566msec

Run status group 6 (all jobs):
   READ: bw=67.6MiB/s (70.9MB/s), 67.6MiB/s-67.6MiB/s (70.9MB/s-70.9MB/s), io=5120MiB (5369MB), run=75766-75766msec

Run status group 7 (all jobs):
  WRITE: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=5120MiB (5369MB), run=20741-20741msec

Disk stats (read/write):
    md0: ios=13148161/13160285, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=3287040/3287278, aggrmerge=0/4456, aggrticks=228488/41028, aggrin_queue=141592, aggrutil=31.53%
  nvme0n1: ios=3287041/3287313, merge=0/4792, ticks=189212/40880, in_queue=105540, util=19.69%
  nvme3n1: ios=3287040/3287270, merge=0/4245, ticks=347392/41592, in_queue=252180, util=31.53%
  nvme2n1: ios=3287040/3287242, merge=0/4263, ticks=189940/40684, in_queue=104764, util=19.60%
  nvme1n1: ios=3287040/3287287, merge=0/4527, ticks=187408/40956, in_queue=103884, util=19.58%