mazgi/console.md

Last active November 19, 2021 18:49

Star (1) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/mazgi/8d0de01f0e2a954968aa.js"></script>
Save mazgi/8d0de01f0e2a954968aa to your computer and use it in GitHub Desktop.

Download ZIP

fio on btrfs raid

Raw

console.md

bs=4k

fio -rw=read -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
fio -rw=write -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
fio -rw=randread -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
fio -rw=randwrite -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite

bs=16k

fio -rw=read -bs=16k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
fio -rw=write -bs=16k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
fio -rw=randread -bs=16k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
fio -rw=randwrite -bs=16k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite

Raw

pat_1.md

# uname -a
Linux Kilchoman 4.1.12-gentoo #1 SMP Mon Dec 28 22:37:32 2015 x86_64 Intel(R) Pentium(R) CPU G4500T @ 3.00GHz GenuineIntel GNU/Linux
# mkfs.btrfs -f /dev/sd{a,b}
btrfs-progs v4.0.1
See http://btrfs.wiki.kernel.org for more information.

Turning ON incompat feature 'extref': increased hardlink limit per file to 65536
Turning ON incompat feature 'skinny-metadata': reduced-size metadata extent refs
adding device /dev/sdb id 2
fs created label (null) on /dev/sda
        nodesize 16384 leafsize 16384 sectorsize 4096 size 7.28TiB
# mount -o device=/dev/sda,device=/dev/sdb /dev/sda /mnt/btrfs
# mount | grep /mnt/btrfs
/dev/sda on /mnt/btrfs type btrfs (rw,relatime,space_cache)
# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   3.7T  0 disk /mnt/btrfs
sdb           8:16   0   3.7T  0 disk 
sr0          11:0    1  1024M  0 rom  
nvme0n1     259:0    0 238.5G  0 disk 
|-nvme0n1p1 259:1    0     4M  0 part 
|-nvme0n1p2 259:2    0     2G  0 part [SWAP]
`-nvme0n1p3 259:3    0    32G  0 part /var/log

Author

mazgi commented Dec 29, 2015

[root@Kilchoman] # cd /mnt/btrfs
[root@Kilchoman] # fio -rw=read -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
seqread: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.2.9
Starting 40 processes
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
seqread: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 40 (f=40): [R(40)] [100.0% done] [23424KB/0KB/0KB /s] [5856/0/0 iops] [eta 00m:00s]
seqread: (groupid=0, jobs=40): err= 0: pid=4212: Tue Dec 29 13:50:47 2015
  read : io=1400.8MB, bw=23765KB/s, iops=5941, runt= 60354msec
    slat (usec): min=2, max=252, avg= 5.41, stdev= 9.43
    clat (usec): min=57, max=2545.1K, avg=214744.88, stdev=198295.08
     lat (usec): min=63, max=2545.5K, avg=214750.30, stdev=198296.21
    clat percentiles (usec):
     |  1.00th=[  149],  5.00th=[  251], 10.00th=[  358], 20.00th=[  506],
     | 30.00th=[  636], 40.00th=[146432], 50.00th=[224256], 60.00th=[272384],
     | 70.00th=[350208], 80.00th=[415744], 90.00th=[473088], 95.00th=[509952],
     | 99.00th=[610304], 99.50th=[643072], 99.90th=[1286144], 99.95th=[1417216],
     | 99.99th=[2310144]
    bw (KB  /s): min=    3, max= 7325, per=2.53%, avg=600.77, stdev=383.95
    lat (usec) : 100=0.19%, 250=4.69%, 500=14.55%, 750=15.01%, 1000=0.60%
    lat (msec) : 2=0.94%, 4=0.06%, 10=0.36%, 20=0.53%, 50=0.53%
    lat (msec) : 100=0.80%, 250=17.00%, 500=38.76%, 750=5.69%, 1000=0.09%
    lat (msec) : 2000=0.17%, >=2000=0.02%
  cpu          : usr=0.03%, sys=0.09%, ctx=45705, majf=0, minf=1588
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=358576/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=1400.8MB, aggrb=23764KB/s, minb=23764KB/s, maxb=23764KB/s, mint=60354msec, maxt=60354msec

Author

mazgi commented Dec 29, 2015

[root@Kilchoman] # cd /mnt/btrfs
[root@Kilchoman] # fio -rw=write -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.2.9
Starting 40 processes
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
seqwrite: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 40 (f=40): [W(40)] [100.0% done] [0KB/41472KB/0KB /s] [0/10.4K/0 iops] [eta 00m:00s]
seqwrite: (groupid=0, jobs=40): err= 0: pid=4319: Tue Dec 29 13:53:59 2015
  write: io=2463.4MB, bw=41923KB/s, iops=10480, runt= 60168msec
    slat (usec): min=7, max=181298, avg=20.76, stdev=864.76
    clat (usec): min=62, max=2402.6K, avg=121926.32, stdev=115685.45
     lat (usec): min=80, max=2402.6K, avg=121947.09, stdev=115714.08
    clat percentiles (usec):
     |  1.00th=[  157],  5.00th=[  209], 10.00th=[  266], 20.00th=[  318],
     | 30.00th=[  438], 40.00th=[123392], 50.00th=[156672], 60.00th=[175104],
     | 70.00th=[187392], 80.00th=[199680], 90.00th=[230400], 95.00th=[242688],
     | 99.00th=[309248], 99.50th=[456704], 99.90th=[1236992], 99.95th=[1482752],
     | 99.99th=[2088960]
    bw (KB  /s): min=    3, max=34728, per=2.54%, avg=1063.01, stdev=743.11
    lat (usec) : 100=0.02%, 250=8.14%, 500=22.89%, 750=0.88%, 1000=0.31%
    lat (msec) : 2=2.94%, 4=0.08%, 10=0.45%, 20=0.33%, 50=0.39%
    lat (msec) : 100=1.49%, 250=58.46%, 500=3.15%, 750=0.10%, 1000=0.22%
    lat (msec) : 2000=0.14%, >=2000=0.02%
  cpu          : usr=0.03%, sys=0.34%, ctx=141225, majf=0, minf=371
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=630608/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=2463.4MB, aggrb=41923KB/s, minb=41923KB/s, maxb=41923KB/s, mint=60168msec, maxt=60168msec

Author

mazgi commented Dec 29, 2015

[root@Kilchoman] # cd /mnt/btrfs
[root@Kilchoman] # fio -rw=randread -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
randread: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.2.9
Starting 40 processes
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
randread: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 40 (f=40): [r(40)] [100.0% done] [2344KB/0KB/0KB /s] [586/0/0 iops] [eta 00m:00s]
randread: (groupid=0, jobs=40): err= 0: pid=4394: Tue Dec 29 14:17:47 2015
  read : io=147984KB, bw=2445.5KB/s, iops=611, runt= 60514msec
    slat (usec): min=3, max=2673.8K, avg=86592.24, stdev=392261.65
    clat (usec): min=4, max=4913.5K, avg=2001004.03, stdev=606256.54
     lat (usec): min=112, max=5732.9K, avg=2087596.28, stdev=728553.02
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   17], 10.00th=[ 1483], 20.00th=[ 2008],
     | 30.00th=[ 2073], 40.00th=[ 2114], 50.00th=[ 2147], 60.00th=[ 2180],
     | 70.00th=[ 2212], 80.00th=[ 2245], 90.00th=[ 2343], 95.00th=[ 2409],
     | 99.00th=[ 2868], 99.50th=[ 4080], 99.90th=[ 4883], 99.95th=[ 4883],
     | 99.99th=[ 4883]
    bw (KB  /s): min=    1, max=  388, per=2.48%, avg=60.71, stdev=26.09
    lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 250=0.01%, 500=0.01%
    lat (usec) : 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=1.15%, 10=2.68%, 20=1.44%, 50=0.37%
    lat (msec) : 100=0.07%, 250=0.17%, 500=0.23%, 750=0.85%, 1000=0.96%
    lat (msec) : 2000=11.11%, >=2000=80.92%
  cpu          : usr=0.00%, sys=0.01%, ctx=3548, majf=0, minf=1582
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=98.3%, >=64=0.0%
     submit    : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=36996/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=147984KB, aggrb=2445KB/s, minb=2445KB/s, maxb=2445KB/s, mint=60514msec, maxt=60514msec

Author

mazgi commented Dec 29, 2015

[root@Kilchoman] # cd /mnt/btrfs
[root@Kilchoman] # fio -rw=randwrite -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.2.9
Starting 40 processes
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
randwrite: Laying out IO file(s) (1 file(s) / 100MB)
Jobs: 40 (f=40): [w(40)] [100.0% done] [0KB/2328KB/0KB /s] [0/582/0 iops] [eta 00m:00s]
randwrite: (groupid=0, jobs=40): err= 0: pid=4452: Tue Dec 29 14:25:44 2015
  write: io=149660KB, bw=2473.2KB/s, iops=618, runt= 60515msec
    slat (usec): min=9, max=2586.2K, avg=99377.33, stdev=421597.78
    clat (usec): min=4, max=4803.8K, avg=1964514.07, stdev=623541.92
     lat (usec): min=187, max=6266.7K, avg=2063891.41, stdev=754293.58
    clat percentiles (usec):
     |  1.00th=[  354],  5.00th=[ 1224], 10.00th=[1548288], 20.00th=[1892352],
     | 30.00th=[1974272], 40.00th=[2039808], 50.00th=[2072576], 60.00th=[2113536],
     | 70.00th=[2179072], 80.00th=[2244608], 90.00th=[2342912], 95.00th=[2408448],
     | 99.00th=[3588096], 99.50th=[3784704], 99.90th=[4227072], 99.95th=[4227072],
     | 99.99th=[4816896]
    bw (KB  /s): min=    1, max=  276, per=2.49%, avg=61.53, stdev=31.57
    lat (usec) : 10=0.03%, 20=0.01%, 50=0.04%, 100=0.05%, 250=0.10%
    lat (usec) : 500=3.51%, 750=0.55%, 1000=0.38%
    lat (msec) : 2=0.69%, 4=0.09%, 10=0.09%, 20=0.13%, 50=0.19%
    lat (msec) : 100=0.09%, 250=0.13%, 500=0.37%, 750=1.31%, 1000=0.63%
    lat (msec) : 2000=24.23%, >=2000=67.38%
  cpu          : usr=0.00%, sys=0.02%, ctx=4150, majf=0, minf=342
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=98.5%, >=64=0.0%
     submit    : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=37415/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=149660KB, aggrb=2473KB/s, minb=2473KB/s, maxb=2473KB/s, mint=60515msec, maxt=60515msec