tmp.txt

basilfx:/tmp/ $ fio --randrepeat=1 --ioengine=libaio --direct=1 --name=testingtons --filename=5GBtestfile --bssplit=512/10:4k/60:8k/20:64k/10 --iodepth=64 --size=5G --readwrite=randrw --rwmixread=75
testingtons: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.2.10
Starting 1 process
testingtons: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [27434KB/8754KB/0KB /s] [6773/2281/0 iops] [eta 00m:00s]
testingtons: (groupid=0, jobs=1): err= 0: pid=18494: Sat Oct  8 11:04:04 2016
  read : io=3840.5MB, bw=37328KB/s, iops=6506, runt=105353msec
    slat (usec): min=1, max=9137, avg= 7.09, stdev=20.97
    clat (usec): min=337, max=1557.3K, avg=8176.76, stdev=14901.69
     lat (usec): min=341, max=1557.3K, avg=8184.05, stdev=14901.65
    clat percentiles (usec):
     |  1.00th=[ 1896],  5.00th=[ 2832], 10.00th=[ 3408], 20.00th=[ 4320],
     | 30.00th=[ 5152], 40.00th=[ 5984], 50.00th=[ 6880], 60.00th=[ 7968],
     | 70.00th=[ 9280], 80.00th=[10944], 90.00th=[13632], 95.00th=[16192],
     | 99.00th=[22912], 99.50th=[27008], 99.90th=[95744], 99.95th=[136192],
     | 99.99th=[272384]
    bw (KB  /s): min= 3137, max=60693, per=100.00%, avg=37827.63, stdev=9340.89
  write: io=1279.6MB, bw=12437KB/s, iops=2169, runt=105353msec
    slat (usec): min=2, max=9868, avg= 8.86, stdev=42.16
    clat (usec): min=139, max=1538.1K, avg=4939.41, stdev=12951.82
     lat (usec): min=143, max=1538.1K, avg=4948.50, stdev=12951.84
    clat percentiles (usec):
     |  1.00th=[  366],  5.00th=[  636], 10.00th=[  908], 20.00th=[ 1448],
     | 30.00th=[ 2096], 40.00th=[ 2832], 50.00th=[ 3568], 60.00th=[ 4448],
     | 70.00th=[ 5536], 80.00th=[ 7072], 90.00th=[ 9664], 95.00th=[12480],
     | 99.00th=[20352], 99.50th=[26240], 99.90th=[93696], 99.95th=[132096],
     | 99.99th=[268288]
    bw (KB  /s): min= 1231, max=21365, per=100.00%, avg=12602.26, stdev=3178.49
    lat (usec) : 250=0.05%, 500=0.64%, 750=1.08%, 1000=1.20%
    lat (msec) : 2=5.17%, 4=17.84%, 10=52.83%, 20=19.49%, 50=1.48%
    lat (msec) : 100=0.13%, 250=0.07%, 500=0.01%, 2000=0.01%
  cpu          : usr=2.76%, sys=8.95%, ctx=521276, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=685481/w=228538/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3840.5MB, aggrb=37327KB/s, minb=37327KB/s, maxb=37327KB/s, mint=105353msec, maxt=105353msec
  WRITE: io=1279.6MB, aggrb=12436KB/s, minb=12436KB/s, maxb=12436KB/s, mint=105353msec, maxt=105353msec

Disk stats (read/write):
    dm-0: ios=685493/229138, merge=0/0, ticks=5585356/1134388, in_queue=6722168, util=99.92%, aggrios=679673/228191, aggrmerge=5916/1005, aggrticks=5544380/1130284, aggrin_queue=6676984, aggrutil=99.90%
  sde: ios=679673/228191, merge=5916/1005, ticks=5544380/1130284, in_queue=6676984, util=99.90%

zfs.txt

basilfx:media/ $ fio --randrepeat=1 --ioengine=libaio --direct=1 --name=testingtons --filename=5GBtestfile --bssplit=512/10:4k/60:8k/20:64k/10 --iodepth=64 --size=5G --readwrite=randrw --rwmixread=75
testingtons: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.2.10
Starting 1 process
testingtons: Laying out IO file(s) (1 file(s) / 5120MB)
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT
fio: pid=20580, err=22/file:filesetup.c:615, func=open(5GBtestfile), error=Invalid argument


Run status group 0 (all jobs):