Skip to content

Instantly share code, notes, and snippets.

@paul

paul/iodata2.mkd

Created August 25, 2010 05:01
Show Gist options
  • Save paul/548888 to your computer and use it in GitHub Desktop.
Save paul/548888 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
iodata2.mkd

Setup

mdadm --create /dev/md0 --level=stripe --assume-clean --chunk=256 --raid-devices=4 /dev/sdd*
mkfs.xfs /dev/md0
mkdir -p /data
mount /dev/md0 /data

iozone

ubuntu@ip-10-204-213-17:/data$ time sudo iozone -a -c -e -s 16g -q 4096 -y 1024
  Iozone: Performance Test of File I/O
          Version $Revision: 3.308 $
    Compiled for 64 bit mode.
    Build: linux

  Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
               Al Slater, Scott Rhine, Mike Wisner, Ken Goss
               Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
               Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
               Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
               Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

  Run began: Tue Aug 24 00:35:01 2010

  Auto Mode
  Include close in write timing
  Include fsync in write timing
  File size set to 16777216 KB
  Using Maximum Record Size 4096 KB
  Using Minimum Record Size 1024 KB
  Command line used: iozone -a -c -e -s 16g -q 4096 -y 1024
  Output is in Kbytes/sec
  Time Resolution = 0.000001 seconds.
  Processor cache size set to 1024 Kbytes.
  Processor cache line size set to 32 bytes.
  File stride size set to 17 * record size.

RAID levels & number of devices

RAID striping over 4 devices seems to be fastest. 8 devices might add some slight overhead.

mdadm raid 10 (4 devices, us-east-1c)

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216    1024   54293   54910    28073    27754   26531   53667   34226  3866759    46332    54295    54641   26995    26992
        16777216    2048   55093   54986    27691    28503   25300   54101   46232  3514783    43014    53342    54351   27226    27064
        16777216    4096   14079   34685    27733    27363   25167   53873   26931  1185901    39912    53763    54453   26688    26471

iozone test complete.

real  317m57.650s
user    0m15.360s
sys      3m7.940s

mdadm raid0 (4 devices, us-east-1c)

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216      64  107447  107670    26663    27847    8171  100583   37547  3414252     5916   106809   107368   27468    28814
        16777216     128  109008  106691    28313    28529   11909  102555   33010  3685793     7401    94513   102277   29750    27722
        16777216     256   94994  100046    27836    28265   15295   53818   52153  3920830     9251    96564   106941   28339    27709
        16777216     512  103140  106661    28786    30306   19510  103628   35090  3948672    14351   103156   102641   28636    29631
        16777216    1024  105420  106845    29217    29140   24037   99795   35943  3958700    41793   100904   104756   28314    28388
        16777216    2048  102313  103988    29509    29526   32999  104148   47435  3676246    38253   104843   102477   29912    28740
        16777216    4096  104454  106390    30620    29130   26533  103748   36379  1217010    36835    99374   103651   27620    28722
        16777216    8192  104111  103621    30635    30385   29068  101222   35731  1057360    37759   101241   100372   28967    29739

iozone test complete.

real  834m12.138s
user    0m30.240s
sys     7m48.070s

mdadm raid0 (8 devices, us-east-1c)

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216    1024  102941  108963    27449    27657   13149  110642   31129  3947665    12101   111456   108874   27253    27293
        16777216    2048  111433  109544    27314    26693   12005   94296   17755  3768964    35679   110979   108590   27361    27562
        16777216    4096  110709  109907    28331    27395   17719  109814   19292  1162884    37610    74501    73700   27128    27043

iozone test complete.

real 340m04.858s
user   0m15.840s
sys    3m46.610s

By zone, RAID stripe with 4 devices

us-east-1d appears to be much faster at writes than the others.

mdadm raid0 (4 devices, us-east-1c)

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216      64  107447  107670    26663    27847    8171  100583   37547  3414252     5916   106809   107368   27468    28814
        16777216     128  109008  106691    28313    28529   11909  102555   33010  3685793     7401    94513   102277   29750    27722
        16777216     256   94994  100046    27836    28265   15295   53818   52153  3920830     9251    96564   106941   28339    27709
        16777216     512  103140  106661    28786    30306   19510  103628   35090  3948672    14351   103156   102641   28636    29631
        16777216    1024  105420  106845    29217    29140   24037   99795   35943  3958700    41793   100904   104756   28314    28388
        16777216    2048  102313  103988    29509    29526   32999  104148   47435  3676246    38253   104843   102477   29912    28740
        16777216    4096  104454  106390    30620    29130   26533  103748   36379  1217010    36835    99374   103651   27620    28722
        16777216    8192  104111  103621    30635    30385   29068  101222   35731  1057360    37759   101241   100372   28967    29739

iozone test complete.

real  834m12.138s
user    0m30.240s
sys     7m48.070s

mdadm raid0 (4 devices, us-east-1d)

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216    1024  100757  106372    93332    90802   58633  104520   81958   813347   107269   104676    93565   93105    92043
        16777216    2048  105501  104566    92072    89694   79010  103678   89517   828620   116193    92056    99196   92439    89983
        16777216    4096  100835  102470    94010    90348   83572  105633  103926   940608   121059   104876   102390   92541    92524
        16777216    8192  107105  105735    85037    85124   91823  105232  109129   947137   123812   107499   105122   91267    91209

iozone test complete.

real  153m14.652s
user    0m32.760s
sys    10m58.070s

mdadm raid0 (4 devices, us-west-1b)

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216    1024  108832  109814    26289    26916   17385  108288   24129  3955086    37746   108981   108434   26363    26361
        16777216    2048  109639  109004    27948    26473   23430  104898   28800  3467243    33853   109171   108628   26606    27057
        16777216    4096  109287  109689    27126    27922   24618  109515   29589  1217741    33390   108471   109093   27105    29088
        16777216    8192  108017  108860    27630    27710   26821  109397   30563  1063123    35220   109090   109083   26985    27291

iozone test complete.

real  377m3.760s
user  0m26.960s
sys 4m40.850s

LVM span over mdadm stripe

Note: LVM spanning doesnt appear to affect read/write performance any.

mdadm raid0 (4 devices, us-east-1d)

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216    1024  100757  106372    93332    90802   58633  104520   81958   813347   107269   104676    93565   93105    92043
        16777216    2048  105501  104566    92072    89694   79010  103678   89517   828620   116193    92056    99196   92439    89983
        16777216    4096  100835  102470    94010    90348   83572  105633  103926   940608   121059   104876   102390   92541    92524
        16777216    8192  107105  105735    85037    85124   91823  105232  109129   947137   123812   107499   105122   91267    91209

iozone test complete.

real  153m14.652s
user    0m32.760s
sys    10m58.070s

lvm span over mdadm raid0 (4 devices, us-east-1d) (same instance & drives as previous)

Setup

modprobe dm_mod
pvcreate /dev/md0
vgcreate ebstest /dev/md0
lvcreate --size=30G --name data --zero y ebstest
mkfs.xfs /dev/ebstest/data

Results

                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        16777216    1024   98186  105383    90419    87595   48989  105347   71171   846502   110599   102741   103810   92771    88901
        16777216    2048  101842  104294    89966    86993   69530  103526  106041   935708   121479   104436   104358   86001    91120
        16777216    4096  101867   97446    88777    87253   79932  105423  100032   925187   115273   104044   103058   90432    88137
        16777216    8192  105516  104693    89293    84666   90347  105217  103650   840287   114191   104178   102031   89186    87854

iozone test complete.

real  155m59.737s
user  0m39.250s
sys 11m21.190s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment