Skip to content

Instantly share code, notes, and snippets.

@thejhh
Last active July 28, 2018 10:49
Show Gist options
  • Select an option

  • Save thejhh/66ff4ae6d2a0a0218ef7353ce4d6bffc to your computer and use it in GitHub Desktop.

Select an option

Save thejhh/66ff4ae6d2a0a0218ef7353ce4d6bffc to your computer and use it in GitHub Desktop.
zpool logs and cache on zvol

Hardware

  • Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
  • 64 G RAM
  • 2x 512 G NVME SSD
  • 2x 10 TB SATA

General setup

  • Swap is 24 GB software RAID-1 md1 (1st partition of NWME)
  • /boot is 512 MB EXT3 on software RAID-1 md0 (2nd partition of NWME)
  • / is Ubuntu 18.04 on 24 GB ZFS root partition (3rd partition of NWME)

First setup: ZFS SLOG and CACHE on ZVOL

# zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 1.38G in 0h0m with 0 errors on Fri Jul 27 14:56:53 2018
config:

        NAME                                                 STATE     READ WRITE CKSUM
        rpool                                                ONLINE       0     0     0
          mirror-0                                           ONLINE       0     0     0
            /dev/disk/by-id/nvme-eui.002538c181b06801-part3  ONLINE       0     0     0
            /dev/disk/by-id/nvme-eui.002538c181b06806-part3  ONLINE       0     0     0

errors: No known data errors

  pool: spool1
 state: ONLINE
  scan: none requested
config:

        NAME                                               STATE     READ WRITE CKSUM
        spool1                                             ONLINE       0     0     0
          /dev/disk/by-id/nvme-eui.002538c181b06806-part4  ONLINE       0     0     0

errors: No known data errors

  pool: spool2
 state: ONLINE
  scan: none requested
config:

        NAME                                               STATE     READ WRITE CKSUM
        spool2                                             ONLINE       0     0     0
          /dev/disk/by-id/nvme-eui.002538c181b06801-part4  ONLINE       0     0     0

errors: No known data errors

  pool: zpool
 state: ONLINE
  scan: none requested
config:

        NAME                                              STATE     READ WRITE CKSUM
        zpool                                             ONLINE       0     0     0
          mirror-0                                        ONLINE       0     0     0
            /dev/disk/by-id/wwn-0x5000c500b0c19e30-part1  ONLINE       0     0     0
            /dev/disk/by-id/wwn-0x5000c500b0c1aad4-part1  ONLINE       0     0     0
        logs
          mirror-1                                        ONLINE       0     0     0
            /dev/zvol/spool1/zpool-slog-1-part1           ONLINE       0     0     0
            /dev/zvol/spool2/zpool-slog-2-part1           ONLINE       0     0     0
        cache
          /dev/zvol/spool1/zpool-cache-1                  ONLINE       0     0     0
          /dev/zvol/spool2/zpool-cache-2                  ONLINE       0     0     0

errors: No known data errors

Test results

Test results with SLOG and CACHE on ZVOL

# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
rpool                 1.42G  21.7G    96K  /
rpool/ROOT            1.32G  21.7G    96K  none
rpool/ROOT/ubuntu     1.32G  21.7G  1.30G  /
rpool/backup           152K  21.7G    96K  /backup
rpool/home             888K  21.7G   108K  /home
rpool/home/root        660K  21.7G   136K  /root
rpool/srv              152K  21.7G    96K  /srv
rpool/var             87.7M  21.7G    96K  /var
rpool/var/cache       68.9M  21.7G  68.9M  /var/cache
rpool/var/log         18.2M  21.7G  3.04M  legacy
rpool/var/mail         152K  21.7G    96K  /var/mail
rpool/var/spool        200K  21.7G   120K  /var/spool
rpool/var/tmp          120K  21.7G   120K  legacy
spool1                21.7G   393G    24K  /spool1
spool1/zpool-cache-1   389M   393G   364M  -
spool1/zpool-slog-1   21.3G   412G  2.57G  -
spool2                21.7G   393G    24K  /spool2
spool2/zpool-cache-2   389M   393G   364M  -
spool2/zpool-slog-2   21.3G   412G  2.57G  -
zpool                 72.3G  8.71T    96K  /zpool
zpool/test            72.0G  8.71T  13.8G  /zpool/test
# time bonnie++ -d bonnie/ -s 1024G -n 0 -m TEST -f -b -u jhh
Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
TEST          1024G           1330380  80 577493  69           2061335  77 322.7  35
Latency                         104ms     388ms               161ms     591ms

1.97,1.97,TEST,1,1532728953,1024G,,,,1330380,80,577493,69,,,2061335,77,322.7,35,,,,,,,,,,,,,,,,,,,104ms,388ms,,161ms,591ms,,,,,,

real    53m35.699s
user    1m39.283s
sys     37m22.875s
Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
TEST1024G13303808057749369206133577322.735
Latency104ms388ms161ms591msLatency
# time bonnie++ -d bonnie/ -s 1024G -n 16 -m TEST -f -b -u jhh                                                   
Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done. 
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
TEST          1024G           1284610  78 573617  68           2053432  77 333.5  37
Latency                         101ms     499ms               138ms    1043ms
Version  1.97       ------Sequential Create------ --------Random Create--------
TEST                -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 16017  49 +++++ +++ 11639  38 15523  49 +++++ +++ 15950  49
Latency              1878us     658us     309ms    1409us      23us    1258us
1.97,1.97,TEST,1,1532741832,1024G,,,,1284610,78,573617,68,,,2053432,77,333.5,37,16,,,,,16017,49,+++++,+++,11639,38,15523,49,+++++,+++,
15950,49,,101ms,499ms,,138ms,1043ms,1878us,658us,309ms,1409us,23us,1258us

real    54m23.006s
user    1m40.360s
sys     37m17.166s
Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
TEST1024G12846107857361768205343277333.537161601749++++++++11639381552349++++++++1595049
Latency101ms499ms138ms1043msLatency1878us658us309ms1409us23us1258us

Second setup: ZFS SLOG and CACHE on partitions

Otherwise same, but:

  • zpool SLOG on NVME partition #5
  • zpool CACHE on NVME partiton #6
Device         Boot     Start        End   Sectors   Size Id Type
/dev/nvme0n1p4      101713920 1000215215 898501296 428.4G  5 Extended
/dev/nvme0n1p5      101715968  143659007  41943040    20G 83 Linux
/dev/nvme0n1p6      143661056 1000215215 856554160 408.4G 83 Linux
# zpool status zpool
  pool: zpool
 state: ONLINE
  scan: none requested
config:

        NAME                                                 STATE     READ WRITE CKSUM
        zpool                                                ONLINE       0     0     0
          mirror-0                                           ONLINE       0     0     0
            /dev/disk/by-id/wwn-0x5000c500b0c19e30-part1     ONLINE       0     0     0
            /dev/disk/by-id/wwn-0x5000c500b0c1aad4-part1     ONLINE       0     0     0
        logs
          mirror-1                                           ONLINE       0     0     0
            /dev/disk/by-id/nvme-eui.002538c181b06801-part5  ONLINE       0     0     0
            /dev/disk/by-id/nvme-eui.002538c181b06806-part5  ONLINE       0     0     0
        cache
          /dev/disk/by-id/nvme-eui.002538c181b06801-part6    ONLINE       0     0     0
          /dev/disk/by-id/nvme-eui.002538c181b06806-part6    ONLINE       0     0     0

errors: No known data errors

Test results with SLOG and CACHE on partitions

# time bonnie++ -d bonnie/ -s 256G -n 16 -m TEST -f -b -u jhh
Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
TEST           256G           1443011  83 1226521  94           3544496  99  9670 682
Latency                         101ms     103ms               344us   36726us
Version  1.97       ------Sequential Create------ --------Random Create--------
TEST                -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   555  10 +++++ +++   529  12   544  10 +++++ +++   541  11
Latency             10900us     638us   10511us   14508us      21us   10643us
1.97,1.97,TEST,1,1532763127,256G,,,,1443011,83,1226521,94,,,3544496,99,9670,682,16,,,,,555,10,+++++,+++,529,12,544,10,+++++,+++,541,11,,101ms,103ms,,344us,36726us,10900us,638us,10511us,14508us,21us,10643us

real    10m9.211s
user    0m19.820s
sys     7m13.818s
Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
TEST256G14430118312265219435444969996706821655510++++++++5291254410++++++++54111
Latency101ms103ms344us36726usLatency10900us638us10511us14508us21us10643us

Second setup: ZFS without SLOG and CACHE

Otherwise same, but zpool has no SLOG or CACHE.

Test results without SLOG and CACHE

# time bonnie++ -d bonnie/ -s 256G -n 16 -m TEST -f -b -u jhh                                                    Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
TEST           256G           1386494  80 1183145  93           3587561  99  1882 160
Latency                         104ms     100ms               823us     159ms
Version  1.97       ------Sequential Create------ --------Random Create--------
TEST                -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   104   3 +++++ +++   101   3   104   3 +++++ +++    98   3
Latency               119ms     704us     115ms     101ms      19us     130ms
1.97,1.97,TEST,1,1532758208,256G,,,,1386494,80,1183145,93,,,3587561,99,1882,160,16,,,,,104,3,+++++,+++,101,3,104,3,+++++,+++,98,3,,104ms,100ms,,823us,159ms,119ms,704us,115ms,101ms,19us,130ms

real    19m6.612s
user    0m20.388s
sys     7m28.235s
Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
TEST256G1386494801183145933587561991882160161043++++++++10131043++++++++983
Latency104ms100ms823us159msLatency119ms704us115ms101ms19us130ms

Results

Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ZVOL#11024G13303808057749369206133577322.735
Latency104ms388ms161ms591msLatency
ZVOL#21024G12846107857361768205343277333.537161601749++++++++11639381552349++++++++
LatencyLatency
PARTITIONS256G14430118312265219435444969996706821655510++++++++5291254410++++++++54111
Latency101ms103ms344us36726usLatency10900us638us10511us14508us21us10643us
WITHOUT256G1386494801183145933587561991882160161043++++++++10131043++++++++983
Latency104ms100ms823us159msLatency119ms704us115ms101ms19us130ms
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment