mazgi/00.README.md

Last active April 5, 2020 11:36

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/mazgi/3a68e5e336194628f21c0ce6e93d38fb.js"></script>
Save mazgi/3a68e5e336194628f21c0ce6e93d38fb to your computer and use it in GitHub Desktop.

Download ZIP

fio benchmark results for btrfs raid

Raw

00.README.md

bs=4k

fio -rw=read -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
fio -rw=write -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
fio -rw=randread -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
fio -rw=randwrite -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite

bs=32m

fio -rw=read -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
fio -rw=write -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
fio -rw=randread -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
fio -rw=randwrite -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite

Devices

blantons /mnt/btrfs # ls -1 /dev/sd* | xargs -I{} smartctl --info {}
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.97-gentoo-x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     SiliconMotion based SSDs
Device Model:     CT480BX200SSD1
Serial Number:    1548F0117189
LU WWN Device Id: 5 00a075 1f0117189
Firmware Version: MU01.4
User Capacity:    480,103,981,056 bytes [480 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Apr  5 11:04:43 2020 -00
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.97-gentoo-x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     SiliconMotion based SSDs
Device Model:     CT480BX200SSD1
Serial Number:    1548F0117E4B
LU WWN Device Id: 5 00a075 1f0117e4b
Firmware Version: MU01.4
User Capacity:    480,103,981,056 bytes [480 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Apr  5 11:04:43 2020 -00
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.97-gentoo-x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     HP SSD S700 500GB
Serial Number:    HBSA30020103933
Firmware Version: S0704A1
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Apr  5 11:04:43 2020 -00
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.97-gentoo-x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     HP SSD S700 500GB
Serial Number:    HBSA30020103932
Firmware Version: S0704A1
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Apr  5 11:04:43 2020 -00
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Operations

blantons ~ # ls -al /mnt/btrfs/
total 16
drwxr-xr-x 1 root root  0 Apr  5 11:13 .
drwxr-xr-x 1 root root 20 Apr  5 10:01 ..
blantons ~ # time btrfs balance start -dconvert=raid10 -mconvert=raid1 /mnt/btrfs
Done, had to relocate 3 out of 3 chunks

real	0m0.177s
user	0m0.003s
sys	0m0.084s
blantons ~ # time btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/btrfs
Done, had to relocate 3 out of 3 chunks

real	0m0.137s
user	0m0.000s
sys	0m0.062s

blantons ~ # blkid /dev/sd*
/dev/sda: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="9e9821d4-d2df-40a1-b51c-9339c6b0c7fe" TYPE="btrfs"
/dev/sdb: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="d8e8f9a2-7827-4422-b332-def20fc80b76" TYPE="btrfs"
/dev/sdc: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="d8e8c9f0-6da3-435c-b11b-a878020f18d6" TYPE="btrfs"
/dev/sdd: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="5a8f6f22-4f85-409f-b83b-1dfdec42e274" TYPE="btrfs"
blantons ~ # btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/btrfs
Done, had to relocate 3 out of 3 chunks
blantons ~ # btrfs balance start -dconvert=raid10 -mconvert=raid1 /mnt/btrfs
Done, had to relocate 3 out of 3 chunks
blantons ~ # blkid /dev/sd*
/dev/sda: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="9e9821d4-d2df-40a1-b51c-9339c6b0c7fe" TYPE="btrfs"
/dev/sdb: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="d8e8f9a2-7827-4422-b332-def20fc80b76" TYPE="btrfs"
/dev/sdc: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="d8e8c9f0-6da3-435c-b11b-a878020f18d6" TYPE="btrfs"
/dev/sdd: UUID="11e097ec-9e20-4943-8563-3e47484f1ce7" UUID_SUB="5a8f6f22-4f85-409f-b83b-1dfdec42e274" TYPE="btrfs"

Raw

32m-raid0.md

    blantons /mnt/btrfs # df -h /mnt/btrfs 
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda        1.8T   16G  1.8T   1% /mnt/btrfs
blantons /mnt/btrfs # btrfs filesystem df /mnt/btrfs 
Data, RAID0: total=20.00GiB, used=15.63GiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=32.14MiB
GlobalReserve, single: total=31.53MiB, used=0.00B
blantons /mnt/btrfs # fio -rw=read -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
seqread: (g=0): rw=read, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 6 (f=4): [_(9),R(1),_(1),f(1),_(4),f(1),_(1),R(1),_(9),R(1),_(3),R(1),_(7)] [3.1% done] [6240MB/0KB/0KB /s] [195/0/0 iops] [eta 09m:24s]                                             s]
seqread: (groupid=0, jobs=40): err= 0: pid=9521: Sun Apr  5 10:20:24 2020
  read : io=19200MB, bw=1077.9MB/s, iops=33, runt= 17813msec
    slat (msec): min=11831, max=17588, avg=15751.66, stdev=1512.57
    clat (msec): min=130, max=535, avg=246.76, stdev=83.10
     lat (msec): min=12304, max=17765, avg=15998.42, stdev=1485.80
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  163], 10.00th=[  169], 20.00th=[  192],
     | 30.00th=[  206], 40.00th=[  212], 50.00th=[  227], 60.00th=[  235],
     | 70.00th=[  251], 80.00th=[  262], 90.00th=[  314], 95.00th=[  469],
     | 99.00th=[  537], 99.50th=[  537], 99.90th=[  537], 99.95th=[  537],
     | 99.99th=[  537]
    lat (msec) : 250=67.50%, 500=30.00%, 750=2.50%
  cpu          : usr=0.00%, sys=2.51%, ctx=8688, majf=0, minf=4915543
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=600/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=19200MB, aggrb=1077.9MB/s, minb=1077.9MB/s, maxb=1077.9MB/s, mint=17813msec, maxt=17813msec
blantons /mnt/btrfs # fio -rw=write -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 1 (f=1): [_(1),K(1),_(1),K(1),_(4),K(1),_(11),W(1),_(1),K(1),_(7),K(1),_(1),K(1),_(1),K(1),_(3),K(1),_(1)] [6.9% done] [0KB/1922MB/0KB /s] [0/60/0 iops] [eta 06m:18s]                                                 
seqwrite: (groupid=0, jobs=40): err= 0: pid=9647: Sun Apr  5 10:27:32 2020
  write: io=15360MB, bw=975057KB/s, iops=33, runt= 16131msec
    slat (msec): min=20, max=4492, avg=2022.58, stdev=1390.86
    clat (msec): min=623, max=11940, avg=7551.43, stdev=3184.75
     lat (msec): min=1031, max=16053, avg=9679.83, stdev=3971.00
    clat percentiles (msec):
     |  1.00th=[  627],  5.00th=[  889], 10.00th=[ 1844], 20.00th=[ 5211],
     | 30.00th=[ 5800], 40.00th=[ 6849], 50.00th=[ 7767], 60.00th=[ 9503],
     | 70.00th=[ 9765], 80.00th=[ 9896], 90.00th=[11338], 95.00th=[11600],
     | 99.00th=[11994], 99.50th=[11994], 99.90th=[11994], 99.95th=[11994],
     | 99.99th=[11994]
    lat (msec) : 750=2.81%, 1000=2.81%, 2000=5.62%, >=2000=78.65%
  cpu          : usr=0.29%, sys=0.72%, ctx=202986, majf=21216, minf=172303
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=534/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=15360MB, aggrb=975056KB/s, minb=975056KB/s, maxb=975056KB/s, mint=16131msec, maxt=16131msec
fio: file hash not empty on exit
blantons /mnt/btrfs # fio -rw=randread -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
randread: (g=0): rw=randread, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 19 (f=17): [r(1),_(1),f(1),_(1),r(1),_(1),r(1),_(1),r(5),f(1),r(1),_(3),r(1),_(3),r(3),_(5),r(1),_(2),r(2),_(2),r(1),_(2)] [4.8% done] [4320MB/0KB/0KB /s] [135/0/0 iops] [eta 05m:55s]
randread: (groupid=0, jobs=40): err= 0: pid=9712: Sun Apr  5 10:30:33 2020
  read : io=19200MB, bw=1074.2MB/s, iops=33, runt= 17875msec
    slat (msec): min=12671, max=17633, avg=16089.92, stdev=1514.01
    clat (msec): min=124, max=486, avg=258.91, stdev=83.76
     lat (msec): min=13036, max=17841, avg=16348.83, stdev=1490.80
    clat percentiles (msec):
     |  1.00th=[  125],  5.00th=[  143], 10.00th=[  167], 20.00th=[  188],
     | 30.00th=[  208], 40.00th=[  229], 50.00th=[  247], 60.00th=[  253],
     | 70.00th=[  273], 80.00th=[  314], 90.00th=[  367], 95.00th=[  449],
     | 99.00th=[  486], 99.50th=[  486], 99.90th=[  486], 99.95th=[  486],
     | 99.99th=[  486]
    lat (msec) : 250=52.50%, 500=47.50%
  cpu          : usr=0.00%, sys=2.50%, ctx=8056, majf=0, minf=4915476
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=600/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=19200MB, aggrb=1074.2MB/s, minb=1074.2MB/s, maxb=1074.2MB/s, mint=17875msec, maxt=17875msec
blantons /mnt/btrfs # fio -rw=randwrite -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite
randwrite: (g=0): rw=randwrite, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 2 (f=2): [_(4),K(2),_(2),K(1),_(2),K(1),_(5),K(1),_(4),K(1),_(1),E(1),_(8),w(1),_(1),K(1),w(1),_(1),K(1),_(1)] [7.5% done] [0KB/960.0MB/0KB /s] [0/30/0 iops] [eta 05m:10s]                                   s]                         
randwrite: (groupid=0, jobs=40): err= 0: pid=9758: Sun Apr  5 10:31:13 2020
  write: io=15360MB, bw=1392.1MB/s, iops=48, runt= 11027msec
    slat (msec): min=22, max=3061, avg=850.55, stdev=730.59
    clat (msec): min=4755, max=10859, avg=7270.64, stdev=1756.55
     lat (msec): min=5191, max=11026, avg=8116.80, stdev=1775.81
    clat percentiles (msec):
     |  1.00th=[ 4752],  5.00th=[ 4817], 10.00th=[ 4948], 20.00th=[ 5800],
     | 30.00th=[ 6063], 40.00th=[ 6456], 50.00th=[ 6980], 60.00th=[ 7373],
     | 70.00th=[ 7963], 80.00th=[ 8979], 90.00th=[ 9765], 95.00th=[10683],
     | 99.00th=[10814], 99.50th=[10814], 99.90th=[10814], 99.95th=[10814],
     | 99.99th=[10814]
    lat (msec) : >=2000=90.23%
  cpu          : usr=0.27%, sys=0.33%, ctx=125946, majf=220, minf=1883
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=532/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=15360MB, aggrb=1392.1MB/s, minb=1392.1MB/s, maxb=1392.1MB/s, mint=11027msec, maxt=11027msec
fio: file hash not empty on exit

  

Raw

32m-raid10.md

    blantons /mnt/btrfs # df -h /mnt/btrfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda        913G   17M  895G   1% /mnt/btrfs
blantons /mnt/btrfs # btrfs filesystem df /mnt/btrfs 
Data, RAID10: total=24.00GiB, used=768.00KiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=112.00KiB
GlobalReserve, single: total=16.00MiB, used=0.00B
blantons /mnt/btrfs # fio -rw=read -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
seqread: (g=0): rw=read, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 4 (f=0): [_(11),f(1),_(9),f(1),_(1),f(1),_(1),f(1),_(14)] [100.0% done] [6727MB/0KB/0KB /s] [210/0/0 iops] [eta 00m:00s]                                                                                                         s]                                              
seqread: (groupid=0, jobs=40): err= 0: pid=10048: Sun Apr  5 10:43:53 2020
  read : io=19200MB, bw=1077.7MB/s, iops=33, runt= 17816msec
    slat (msec): min=9331, max=17539, avg=13861.52, stdev=3020.92
    clat (msec): min=110, max=409, avg=194.65, stdev=76.09
     lat (msec): min=9478, max=17691, avg=14056.18, stdev=3051.04
    clat percentiles (msec):
     |  1.00th=[  111],  5.00th=[  115], 10.00th=[  118], 20.00th=[  143],
     | 30.00th=[  147], 40.00th=[  153], 50.00th=[  163], 60.00th=[  188],
     | 70.00th=[  202], 80.00th=[  237], 90.00th=[  318], 95.00th=[  367],
     | 99.00th=[  408], 99.50th=[  408], 99.90th=[  408], 99.95th=[  408],
     | 99.99th=[  408]
    lat (msec) : 250=82.50%, 500=17.50%
  cpu          : usr=0.00%, sys=2.62%, ctx=8715, majf=0, minf=4915504
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=600/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=19200MB, aggrb=1077.7MB/s, minb=1077.7MB/s, maxb=1077.7MB/s, mint=17816msec, maxt=17816msec
blantons /mnt/btrfs # fio -rw=write -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 1 (f=1): [_(2),W(1),_(1),K(1),_(4),K(1),_(1),K(1),_(4),K(2),_(1),K(1),_(2),K(1),_(5),K(1),_(11)] [9.1% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 10m:53s]                                                  s]                   
seqwrite: (groupid=0, jobs=40): err= 0: pid=10095: Sun Apr  5 10:46:41 2020
  write: io=15360MB, bw=300492KB/s, iops=9, runt= 52343msec
    slat (msec): min=20, max=3820, avg=1185.83, stdev=925.94
    clat (msec): min=1780, max=51451, avg=19819.78, stdev=14736.79
     lat (msec): min=2130, max=52341, avg=21005.61, stdev=14761.35
    clat percentiles (msec):
     |  1.00th=[ 1778],  5.00th=[ 2180], 10.00th=[ 3359], 20.00th=[ 6915],
     | 30.00th=[10159], 40.00th=[12780], 50.00th=[14746], 60.00th=[16712],
     | 70.00th=[16712], 80.00th=[16712], 90.00th=[16712], 95.00th=[16712],
     | 99.00th=[16712], 99.50th=[16712], 99.90th=[16712], 99.95th=[16712],
     | 99.99th=[16712]
    lat (msec) : 2000=2.91%, >=2000=90.29%
  cpu          : usr=0.17%, sys=0.25%, ctx=80775, majf=13124, minf=83963
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=515/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=15360MB, aggrb=300491KB/s, minb=300491KB/s, maxb=300491KB/s, mint=52343msec, maxt=52343msec
fio: file hash not empty on exit
blantons /mnt/btrfs # fio -rw=randread -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
randread: (g=0): rw=randread, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 6 (f=5): [_(2),r(1),_(5),f(1),_(3),r(1),_(9),r(1),_(1),r(1),_(11),r(1),_(3)] [3.6% done] [4800MB/0KB/0KB /s] [150/0/0 iops] [eta 07m:56s]                                                                                         s]                                        
randread: (groupid=0, jobs=40): err= 0: pid=10157: Sun Apr  5 10:50:19 2020
  read : io=19200MB, bw=1080.7MB/s, iops=33, runt= 17767msec
    slat (msec): min=9978, max=17439, avg=13546.81, stdev=2776.73
    clat (msec): min=98, max=416, avg=211.19, stdev=64.04
     lat (msec): min=10193, max=17628, avg=13757.99, stdev=2795.11
    clat percentiles (msec):
     |  1.00th=[   98],  5.00th=[  102], 10.00th=[  110], 20.00th=[  155],
     | 30.00th=[  188], 40.00th=[  202], 50.00th=[  210], 60.00th=[  221],
     | 70.00th=[  233], 80.00th=[  255], 90.00th=[  289], 95.00th=[  302],
     | 99.00th=[  416], 99.50th=[  416], 99.90th=[  416], 99.95th=[  416],
     | 99.99th=[  416]
    lat (msec) : 100=2.50%, 250=75.00%, 500=22.50%
  cpu          : usr=0.00%, sys=2.63%, ctx=8435, majf=0, minf=4915490
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=600/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=19200MB, aggrb=1080.7MB/s, minb=1080.7MB/s, maxb=1080.7MB/s, mint=17767msec, maxt=17767msec
blantons /mnt/btrfs # fio -rw=randwrite -bs=32m -size=500m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite
randwrite: (g=0): rw=randwrite, bs=32M-32M/32M-32M/32M-32M, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 2 (f=2): [_(6),K(1),_(1),K(1),_(1),K(1),_(1),K(2),_(4),w(1),_(1),K(1),_(5),K(2),_(1),w(1),_(10)] [6.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 23m:58s]                                                                             
randwrite: (groupid=0, jobs=40): err= 0: pid=13344: Sun Apr  5 10:56:10 2020
  write: io=15360MB, bw=188935KB/s, iops=6, runt= 83249msec
    slat (msec): min=26, max=4351, avg=2309.29, stdev=1299.16
    clat (msec): min=2006, max=81840, avg=46811.47, stdev=24565.13
     lat (msec): min=2397, max=83248, avg=49120.76, stdev=24090.54
    clat percentiles (msec):
     |  1.00th=[ 2008],  5.00th=[ 3687], 10.00th=[11731], 20.00th=[16712],
     | 30.00th=[16712], 40.00th=[16712], 50.00th=[16712], 60.00th=[16712],
     | 70.00th=[16712], 80.00th=[16712], 90.00th=[16712], 95.00th=[16712],
     | 99.00th=[16712], 99.50th=[16712], 99.90th=[16712], 99.95th=[16712],
     | 99.99th=[16712]
    lat (msec) : >=2000=89.72%
  cpu          : usr=0.05%, sys=0.10%, ctx=99795, majf=54, minf=944
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=535/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=15360MB, aggrb=188934KB/s, minb=188934KB/s, maxb=188934KB/s, mint=83249msec, maxt=83249msec
fio: file hash not empty on exit

  

Raw

4k-raid0.md

    blantons /mnt/btrfs # df -h /mnt/btrfs 
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda        1.8T   16G  1.8T   1% /mnt/btrfs
blantons /mnt/btrfs # btrfs filesystem df /mnt/btrfs 
Data, RAID0: total=20.00GiB, used=15.63GiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=32.14MiB
GlobalReserve, single: total=31.53MiB, used=0.00B
blantons /mnt/btrfs # fio -rw=read -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
seqread: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 40 (f=40): [R(40)] [83.3% done] [888.6MB/0KB/0KB /s] [227K/0/0 iops] [eta 00m:01s]
seqread: (groupid=0, jobs=40): err= 0: pid=9122: Sun Apr  5 10:07:23 2020
  read : io=4000.0MB, bw=906596KB/s, iops=226648, runt=  4518msec
    slat (usec): min=2, max=16442, avg= 5.56, stdev=44.95
    clat (usec): min=24, max=33432, avg=5576.25, stdev=5368.93
     lat (usec): min=29, max=33435, avg=5581.81, stdev=5368.07
    clat percentiles (usec):
     |  1.00th=[  298],  5.00th=[  446], 10.00th=[  556], 20.00th=[  692],
     | 30.00th=[  836], 40.00th=[ 1012], 50.00th=[ 1784], 60.00th=[ 7008],
     | 70.00th=[ 7712], 80.00th=[13120], 90.00th=[13888], 95.00th=[14144],
     | 99.00th=[14784], 99.50th=[15040], 99.90th=[23168], 99.95th=[24960],
     | 99.99th=[31616]
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.59%, 500=6.71%, 750=16.72%
    lat (usec) : 1000=15.49%
    lat (msec) : 2=10.55%, 4=0.19%, 10=25.23%, 20=24.22%, 50=0.28%
  cpu          : usr=0.51%, sys=2.64%, ctx=291330, majf=0, minf=1611
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1024000/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4000.0MB, aggrb=906595KB/s, minb=906595KB/s, maxb=906595KB/s, mint=4518msec, maxt=4518msec
blantons /mnt/btrfs # fio -rw=write -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 33 (f=31): [W(2),_(2),W(2),f(1),_(1),W(4),_(1),W(3),f(1),W(13),_(1),W(2),_(1),W(3),_(1),W(2)] [88.9% done] [0KB/392.2MB/0KB /s] [0/100K/0 iops] [eta 00m:01s]
seqwrite: (groupid=0, jobs=40): err= 0: pid=9172: Sun Apr  5 10:08:03 2020
  write: io=4000.0MB, bw=499390KB/s, iops=124847, runt=  8202msec
    slat (usec): min=8, max=103378, avg=288.29, stdev=1366.24
    clat (usec): min=0, max=154533, avg=9682.41, stdev=9506.79
     lat (usec): min=32, max=154553, avg=9970.70, stdev=9645.85
    clat percentiles (usec):
     |  1.00th=[  173],  5.00th=[  318], 10.00th=[  422], 20.00th=[  478],
     | 30.00th=[  828], 40.00th=[ 7456], 50.00th=[11712], 60.00th=[12608],
     | 70.00th=[12992], 80.00th=[13376], 90.00th=[15296], 95.00th=[25984],
     | 99.00th=[46336], 99.50th=[54016], 99.90th=[73216], 99.95th=[80384],
     | 99.99th=[97792]
    lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.12%
    lat (usec) : 100=0.17%, 250=3.42%, 500=19.05%, 750=6.26%, 1000=2.06%
    lat (msec) : 2=1.56%, 4=2.20%, 10=9.67%, 20=48.42%, 50=6.32%
    lat (msec) : 100=0.72%, 250=0.01%
  cpu          : usr=0.50%, sys=5.52%, ctx=881117, majf=0, minf=380
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1024000/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=4000.0MB, aggrb=499390KB/s, minb=499390KB/s, maxb=499390KB/s, mint=8202msec, maxt=8202msec
blantons /mnt/btrfs # fio -rw=randread -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
randread: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 40 (f=40): [r(40)] [88.9% done] [646.2MB/0KB/0KB /s] [165K/0/0 iops] [eta 00m:01s]
randread: (groupid=0, jobs=40): err= 0: pid=9226: Sun Apr  5 10:08:49 2020
  read : io=4000.0MB, bw=527088KB/s, iops=131771, runt=  7771msec
    slat (usec): min=2, max=35632, avg=326.13, stdev=1185.41
    clat (usec): min=3, max=57473, avg=9277.81, stdev=4744.31
     lat (usec): min=65, max=58270, avg=9603.94, stdev=4974.50
    clat percentiles (usec):
     |  1.00th=[  620],  5.00th=[ 2352], 10.00th=[ 3536], 20.00th=[ 5152],
     | 30.00th=[ 6432], 40.00th=[ 7648], 50.00th=[ 8768], 60.00th=[10048],
     | 70.00th=[11328], 80.00th=[13120], 90.00th=[15552], 95.00th=[17792],
     | 99.00th=[22144], 99.50th=[24192], 99.90th=[28288], 99.95th=[30080],
     | 99.99th=[35072]
    lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.36%, 500=0.49%, 750=0.27%, 1000=0.34%
    lat (msec) : 2=2.42%, 4=8.69%, 10=47.30%, 20=37.78%, 50=2.32%
    lat (msec) : 100=0.01%
  cpu          : usr=0.36%, sys=2.30%, ctx=449397, majf=0, minf=1564
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1024000/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4000.0MB, aggrb=527087KB/s, minb=527087KB/s, maxb=527087KB/s, mint=7771msec, maxt=7771msec
blantons /mnt/btrfs # fio -rw=randwrite -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 10 (f=6): [_(1),w(1),_(2),f(1),_(5),f(1),_(4),f(1),w(1),_(4),w(1),_(3),w(1),_(5),w(1),_(6),w(1),f(1)] [100.0% done] [0KB/339.8MB/0KB /s] [0/86.1K/0 iops] [eta 00m:00s]
randwrite: (groupid=0, jobs=40): err= 0: pid=9272: Sun Apr  5 10:09:18 2020
  write: io=4000.0MB, bw=410997KB/s, iops=102749, runt=  9966msec
    slat (usec): min=10, max=111281, avg=362.81, stdev=2237.07
    clat (usec): min=2, max=191905, avg=11317.88, stdev=14077.65
     lat (usec): min=34, max=191918, avg=11680.69, stdev=14309.01
    clat percentiles (usec):
     |  1.00th=[  286],  5.00th=[  466], 10.00th=[  502], 20.00th=[  540],
     | 30.00th=[  636], 40.00th=[ 1672], 50.00th=[ 9408], 60.00th=[13760],
     | 70.00th=[15552], 80.00th=[16320], 90.00th=[24960], 95.00th=[38144],
     | 99.00th=[70144], 99.50th=[80384], 99.90th=[107008], 99.95th=[122368],
     | 99.99th=[154624]
    lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.04%
    lat (usec) : 250=0.63%, 500=9.27%, 750=23.13%, 1000=2.84%
    lat (msec) : 2=4.69%, 4=2.03%, 10=8.71%, 20=35.79%, 50=10.11%
    lat (msec) : 100=2.58%, 250=0.15%
  cpu          : usr=0.40%, sys=5.30%, ctx=613118, majf=0, minf=340
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1024000/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=4000.0MB, aggrb=410997KB/s, minb=410997KB/s, maxb=410997KB/s, mint=9966msec, maxt=9966msec

  

Raw

4k-raid10.md

    blantons /mnt/btrfs # df -h /mnt/btrfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda        913G   17M  895G   1% /mnt/btrfs
blantons /mnt/btrfs # btrfs filesystem df /mnt/btrfs 
Data, RAID10: total=24.00GiB, used=768.00KiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=112.00KiB
GlobalReserve, single: total=16.00MiB, used=0.00B
blantons /mnt/btrfs # fio -rw=read -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqread
seqread: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 20 (f=17): [_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),R(1),_(1),f(1),_(1),R(1),_(1),f(1),_(1),f(1),_(1),R(1),_(1),R(1)] [100.0% done] [471.3MB/0KB/0KB /s] [121K/0/0 iops] [eta 00m:00s]
seqread: (groupid=0, jobs=40): err= 0: pid=9860: Sun Apr  5 10:36:58 2020
  read : io=4000.0MB, bw=825307KB/s, iops=206326, runt=  4963msec
    slat (usec): min=2, max=889, avg= 5.07, stdev=12.48
    clat (usec): min=21, max=21036, avg=5470.14, stdev=4550.88
     lat (usec): min=32, max=21039, avg=5475.21, stdev=4550.68
    clat percentiles (usec):
     |  1.00th=[  378],  5.00th=[  604], 10.00th=[  748], 20.00th=[  980],
     | 30.00th=[ 1192], 40.00th=[ 1464], 50.00th=[ 2512], 60.00th=[ 8256],
     | 70.00th=[ 9024], 80.00th=[10688], 90.00th=[11328], 95.00th=[11712],
     | 99.00th=[13248], 99.50th=[14400], 99.90th=[16192], 99.95th=[17024],
     | 99.99th=[18816]
    lat (usec) : 50=0.01%, 100=0.03%, 250=0.18%, 500=2.61%, 750=7.23%
    lat (usec) : 1000=10.98%
    lat (msec) : 2=27.64%, 4=1.81%, 10=24.45%, 20=25.05%, 50=0.01%
  cpu          : usr=0.55%, sys=2.72%, ctx=252834, majf=0, minf=1656
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1024000/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4000.0MB, aggrb=825307KB/s, minb=825307KB/s, maxb=825307KB/s, mint=4963msec, maxt=4963msec
blantons /mnt/btrfs # fio -rw=write -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=seqwrite
seqwrite: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 21 (f=18): [W(6),_(2),W(1),f(1),_(5),W(2),_(2),W(1),f(1),W(3),f(1),_(4),W(1),_(3),W(1),_(2),W(1),_(1),W(2)] [76.9% done] [0KB/306.7MB/0KB /s] [0/78.5K/0 iops] [eta 00m:03s]
seqwrite: (groupid=0, jobs=40): err= 0: pid=9906: Sun Apr  5 10:37:35 2020
  write: io=4000.0MB, bw=401057KB/s, iops=100264, runt= 10213msec
    slat (usec): min=8, max=51019, avg=162.47, stdev=728.51
    clat (usec): min=0, max=98169, avg=12145.34, stdev=11495.90
     lat (usec): min=36, max=98179, avg=12307.81, stdev=11564.41
    clat percentiles (usec):
     |  1.00th=[  110],  5.00th=[  258], 10.00th=[  358], 20.00th=[  596],
     | 30.00th=[ 1336], 40.00th=[ 7392], 50.00th=[12992], 60.00th=[15168],
     | 70.00th=[16512], 80.00th=[18304], 90.00th=[26496], 95.00th=[34560],
     | 99.00th=[49408], 99.50th=[54016], 99.90th=[63232], 99.95th=[68096],
     | 99.99th=[79360]
    lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.29%, 100=0.58%
    lat (usec) : 250=3.63%, 500=11.24%, 750=8.35%, 1000=3.36%
    lat (msec) : 2=5.47%, 4=3.53%, 10=6.44%, 20=41.18%, 50=15.02%
    lat (msec) : 100=0.91%
  cpu          : usr=0.30%, sys=4.84%, ctx=497270, majf=0, minf=342
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1024000/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=4000.0MB, aggrb=401057KB/s, minb=401057KB/s, maxb=401057KB/s, mint=10213msec, maxt=10213msec
blantons /mnt/btrfs # fio -rw=randread -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randread
randread: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 8 (f=6): [_(6),f(1),_(1),r(1),_(7),r(1),_(1),r(1),_(1),r(1),_(5),f(1),_(9),r(1),_(1),r(1),_(1)] [100.0% done] [595.5MB/0KB/0KB /s] [152K/0/0 iops] [eta 00m:00s]
randread: (groupid=0, jobs=40): err= 0: pid=9955: Sun Apr  5 10:38:42 2020
  read : io=4000.0MB, bw=680173KB/s, iops=170043, runt=  6022msec
    slat (usec): min=2, max=38434, avg=188.68, stdev=668.37
    clat (usec): min=2, max=88514, avg=6783.72, stdev=3880.93
     lat (usec): min=75, max=89075, avg=6972.39, stdev=4015.51
    clat percentiles (usec):
     |  1.00th=[  668],  5.00th=[ 1688], 10.00th=[ 2608], 20.00th=[ 3952],
     | 30.00th=[ 5024], 40.00th=[ 5792], 50.00th=[ 6304], 60.00th=[ 6560],
     | 70.00th=[ 7712], 80.00th=[ 9152], 90.00th=[11328], 95.00th=[13376],
     | 99.00th=[18304], 99.50th=[21632], 99.90th=[40192], 99.95th=[45312],
     | 99.99th=[54528]
    lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
    lat (usec) : 250=0.12%, 500=0.45%, 750=0.66%, 1000=0.80%
    lat (msec) : 2=4.47%, 4=13.99%, 10=63.97%, 20=14.87%, 50=0.66%
    lat (msec) : 100=0.02%
  cpu          : usr=0.60%, sys=3.41%, ctx=586263, majf=0, minf=1607
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1024000/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4000.0MB, aggrb=680172KB/s, minb=680172KB/s, maxb=680172KB/s, mint=6022msec, maxt=6022msec
blantons /mnt/btrfs # fio -rw=randwrite -bs=4k -size=100m -numjobs=40 -runtime=60 -direct=1 -invalidate=1 -ioengine=libaio -iodepth=32 -iodepth_batch=32 -group_reporting -name=randwrite
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.15
Starting 40 processes
Jobs: 34 (f=34): [w(22),_(1),w(4),_(1),w(3),_(1),w(1),_(2),w(3),_(1),w(1)] [86.7% done] [0KB/380.7MB/0KB /s] [0/97.5K/0 iops] [eta 00m:02s]
randwrite: (groupid=0, jobs=40): err= 0: pid=10001: Sun Apr  5 10:39:33 2020
  write: io=4000.0MB, bw=301332KB/s, iops=75332, runt= 13593msec
    slat (usec): min=10, max=159396, avg=518.10, stdev=2916.75
    clat (usec): min=3, max=300901, avg=15975.43, stdev=19470.17
     lat (usec): min=46, max=300918, avg=16493.53, stdev=19855.24
    clat percentiles (usec):
     |  1.00th=[  330],  5.00th=[  548], 10.00th=[  668], 20.00th=[  868],
     | 30.00th=[ 1736], 40.00th=[ 6560], 50.00th=[10048], 60.00th=[13376],
     | 70.00th=[18048], 80.00th=[26752], 90.00th=[42240], 95.00th=[56576],
     | 99.00th=[86528], 99.50th=[98816], 99.90th=[130560], 99.95th=[144384],
     | 99.99th=[220160]
    lat (usec) : 4=0.01%, 10=0.01%, 50=0.01%, 100=0.03%, 250=0.38%
    lat (usec) : 500=3.36%, 750=10.98%, 1000=8.88%
    lat (msec) : 2=7.02%, 4=3.40%, 10=15.94%, 20=23.13%, 50=19.68%
    lat (msec) : 100=6.73%, 250=0.46%, 500=0.01%
  cpu          : usr=0.31%, sys=4.55%, ctx=547576, majf=0, minf=341
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1024000/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: io=4000.0MB, aggrb=301331KB/s, minb=301331KB/s, maxb=301331KB/s, mint=13593msec, maxt=13593msec

  