Using fio from https://github.com/axboe/fio
Tests are run in a zone and the file system been written to has primarycache and secondarycache set to none
zfs get primarycache,secondarycache /ssd/79b49b18-9111-4020-a0a5-2f96364b01e1
NAME PROPERTY VALUE SOURCE
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 primarycache none local
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 secondarycache none local
pool: ssd
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
ssd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t19d1 ONLINE 0 0 0
c0t20d1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c0t21d1 ONLINE 0 0 0
c0t22d1 ONLINE 0 0 0
Dataset properties for reference
zfs get all /ssd/79b49b18-9111-4020-a0a5-2f96364b01e1
NAME PROPERTY VALUE SOURCE
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 type filesystem -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 creation Tue Dec 1 14:54 2015 -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 used 24.0G -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 available 336G -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 referenced 24.0G -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 compressratio 1.00x -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 mounted yes -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 quota none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 reservation none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 recordsize 128K default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 mountpoint /ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 sharenfs off default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 checksum on default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 compression off default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 atime on default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 devices off temporary
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 exec on default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 setuid on default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 readonly off default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 zoned on local
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 snapdir hidden default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 aclmode discard default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 aclinherit restricted default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 canmount on default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 xattr on default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 copies 1 default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 version 5 -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 utf8only off -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 normalization none -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 casesensitivity sensitive -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 vscan off default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 nbmand off default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 sharesmb off default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 refquota none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 refreservation none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 primarycache none local
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 secondarycache none local
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 usedbysnapshots 0 -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 usedbydataset 24.0G -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 usedbychildren 0 -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 usedbyrefreservation 0 -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 logbias latency default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 dedup off default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 mlslabel none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 sync standard default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 refcompressratio 1.00x -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 written 24.0G -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 logicalused 24.0G -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 logicalreferenced 24.0G -
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 filesystem_limit none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 snapshot_limit none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 filesystem_count none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 snapshot_count none default
ssd/79b49b18-9111-4020-a0a5-2f96364b01e1 redundant_metadata all default
SSDs are Intel S3710's http://www.intel.com/content/www/us/en/solid-state-drives/ssd-dc-s3710-spec.html
Controller is a Dell PERC H710
[global]
bs=4k
ioengine=posixaio
iodepth=32
size=8g
runtime=60
fallocate=none
clocksource=clock_gettime
group_reporting=1
numjobs=4
directory=/ssd/79b49b18-9111-4020-a0a5-2f96364b01e1/
filename=ssd.test.file
[seq-read]
rw=read
stonewall
[rand-read]
rw=randread
stonewall
[seq-write]
rw=write
stonewall
[rand-write]
rw=randwrite
stonewall
fio-2.2.12-10-gb9c8
Starting 16 processes
Jobs: 4 (f=4): [_(12),w(4)] [25.5% done] [0KB/12604KB/0KB /s] [0/3151/0 iops] [eta 11m:08s]
seq-read: (groupid=0, jobs=4): err= 0: pid=90460: Tue Dec 1 16:08:02 2015
read : io=32768MB, bw=709246KB/s, iops=177311, runt= 47310msec
slat (usec): min=0, max=668, avg= 5.37, stdev=10.72
clat (usec): min=3, max=4511, avg=648.99, stdev=189.85
lat (usec): min=15, max=4514, avg=654.35, stdev=189.72
clat percentiles (usec):
| 1.00th=[ 141], 5.00th=[ 390], 10.00th=[ 454], 20.00th=[ 524],
| 30.00th=[ 564], 40.00th=[ 604], 50.00th=[ 636], 60.00th=[ 676],
| 70.00th=[ 716], 80.00th=[ 764], 90.00th=[ 844], 95.00th=[ 932],
| 99.00th=[ 1320], 99.50th=[ 1432], 99.90th=[ 1656], 99.95th=[ 1864],
| 99.99th=[ 2992]
bw (KB /s): min=170920, max=180104, per=25.01%, avg=177365.72, stdev=1657.90
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.10%, 100=0.41%
lat (usec) : 250=1.37%, 500=14.33%, 750=61.40%, 1000=19.08%
lat (msec) : 2=3.27%, 4=0.04%, 10=0.01%
cpu : usr=34.98%, sys=499.21%, ctx=17632148, majf=0, minf=0
IO depths : 1=0.1%, 2=0.2%, 4=2.1%, 8=11.2%, 16=77.0%, 32=9.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=94.0%, 8=1.6%, 16=2.5%, 32=1.9%, 64=0.0%, >=64=0.0%
issued : total=r=8388608/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
rand-read: (groupid=1, jobs=4): err= 0: pid=90481: Tue Dec 1 16:08:02 2015
read : io=3109.4MB, bw=53053KB/s, iops=13263, runt= 60014msec
slat (usec): min=0, max=352, avg= 3.10, stdev= 1.32
clat (usec): min=24, max=54320, avg=9632.18, stdev=4795.51
lat (usec): min=26, max=54322, avg=9635.28, stdev=4795.40
clat percentiles (usec):
| 1.00th=[ 1544], 5.00th=[ 2832], 10.00th=[ 3856], 20.00th=[ 5344],
| 30.00th=[ 6688], 40.00th=[ 7904], 50.00th=[ 9152], 60.00th=[10432],
| 70.00th=[11840], 80.00th=[13504], 90.00th=[15680], 95.00th=[17792],
| 99.00th=[23680], 99.50th=[25984], 99.90th=[30848], 99.95th=[33536],
| 99.99th=[38656]
bw (KB /s): min=12400, max=14208, per=25.02%, avg=13276.13, stdev=256.81
lat (usec) : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.07%, 1000=0.18%
lat (msec) : 2=1.79%, 4=8.88%, 10=45.51%, 20=40.75%, 50=2.81%
lat (msec) : 100=0.01%
cpu : usr=2.92%, sys=12.46%, ctx=3010153, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=75.5%, 32=24.4%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.1%, 8=4.8%, 16=2.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=795985/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
seq-write: (groupid=2, jobs=4): err= 0: pid=90511: Tue Dec 1 16:08:02 2015
write: io=32285MB, bw=550991KB/s, iops=137747, runt= 60001msec
slat (usec): min=0, max=1698, avg= 7.69, stdev=23.47
clat (usec): min=1, max=20856, avg=870.72, stdev=991.70
lat (usec): min=23, max=20859, avg=878.41, stdev=991.04
clat percentiles (usec):
| 1.00th=[ 78], 5.00th=[ 167], 10.00th=[ 258], 20.00th=[ 398],
| 30.00th=[ 510], 40.00th=[ 604], 50.00th=[ 684], 60.00th=[ 764],
| 70.00th=[ 860], 80.00th=[ 988], 90.00th=[ 1224], 95.00th=[ 2128],
| 99.00th=[ 5728], 99.50th=[ 6560], 99.90th=[ 9280], 99.95th=[10560],
| 99.99th=[12992]
bw (KB /s): min=130473, max=144608, per=25.02%, avg=137844.02, stdev=3695.72
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.04%, 50=0.20%
lat (usec) : 100=1.56%, 250=7.75%, 500=19.47%, 750=29.58%, 1000=22.29%
lat (msec) : 2=13.98%, 4=2.06%, 10=2.99%, 20=0.07%, 50=0.01%
cpu : usr=31.35%, sys=487.64%, ctx=22241311, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=2.6%, 16=76.7%, 32=20.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.7%, 8=3.3%, 16=2.3%, 32=0.7%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=8265001/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
rand-write: (groupid=3, jobs=4): err= 0: pid=90539: Tue Dec 1 16:08:02 2015
write: io=824504KB, bw=13725KB/s, iops=3431, runt= 60072msec
slat (usec): min=0, max=394, avg= 3.27, stdev= 1.89
clat (usec): min=12, max=255148, avg=37274.35, stdev=25766.55
lat (usec): min=15, max=255149, avg=37277.62, stdev=25766.45
clat percentiles (usec):
| 1.00th=[ 75], 5.00th=[ 4384], 10.00th=[11456], 20.00th=[15296],
| 30.00th=[18048], 40.00th=[21888], 50.00th=[27776], 60.00th=[39680],
| 70.00th=[54016], 80.00th=[64256], 90.00th=[73216], 95.00th=[79360],
| 99.00th=[103936], 99.50th=[116224], 99.90th=[140288], 99.95th=[148480],
| 99.99th=[171008]
bw (KB /s): min= 2601, max= 5880, per=25.04%, avg=3436.53, stdev=517.94
lat (usec) : 20=0.11%, 50=0.59%, 100=0.45%, 250=0.39%, 500=0.46%
lat (usec) : 750=0.30%, 1000=0.24%
lat (msec) : 2=0.92%, 4=1.33%, 10=3.38%, 20=27.47%, 50=31.29%
lat (msec) : 100=31.82%, 250=1.24%, 500=0.01%
cpu : usr=0.79%, sys=4.27%, ctx=727503, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=76.2%, 32=23.7%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.1%, 8=4.7%, 16=2.2%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=206126/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: io=32768MB, aggrb=709246KB/s, minb=709246KB/s, maxb=709246KB/s, mint=47310msec, maxt=47310msec
Run status group 1 (all jobs):
READ: io=3109.4MB, aggrb=53053KB/s, minb=53053KB/s, maxb=53053KB/s, mint=60014msec, maxt=60014msec
Run status group 2 (all jobs):
WRITE: io=32285MB, aggrb=550990KB/s, minb=550990KB/s, maxb=550990KB/s, mint=60001msec, maxt=60001msec
Run status group 3 (all jobs):
WRITE: io=824504KB, aggrb=13725KB/s, minb=13725KB/s, maxb=13725KB/s, mint=60072msec, maxt=60072msec