- Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
- 64 G RAM
- 2x 512 G NVME SSD
- 2x 10 TB SATA
- Swap is 24 GB software RAID-1 md1 (1st partition of NWME)
- /boot is 512 MB EXT3 on software RAID-1 md0 (2nd partition of NWME)
- / is Ubuntu 18.04 on 24 GB ZFS root partition (3rd partition of NWME)
# zpool status
pool: rpool
state: ONLINE
scan: resilvered 1.38G in 0h0m with 0 errors on Fri Jul 27 14:56:53 2018
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
/dev/disk/by-id/nvme-eui.002538c181b06801-part3 ONLINE 0 0 0
/dev/disk/by-id/nvme-eui.002538c181b06806-part3 ONLINE 0 0 0
errors: No known data errors
pool: spool1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
spool1 ONLINE 0 0 0
/dev/disk/by-id/nvme-eui.002538c181b06806-part4 ONLINE 0 0 0
errors: No known data errors
pool: spool2
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
spool2 ONLINE 0 0 0
/dev/disk/by-id/nvme-eui.002538c181b06801-part4 ONLINE 0 0 0
errors: No known data errors
pool: zpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500b0c19e30-part1 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500b0c1aad4-part1 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
/dev/zvol/spool1/zpool-slog-1-part1 ONLINE 0 0 0
/dev/zvol/spool2/zpool-slog-2-part1 ONLINE 0 0 0
cache
/dev/zvol/spool1/zpool-cache-1 ONLINE 0 0 0
/dev/zvol/spool2/zpool-cache-2 ONLINE 0 0 0
errors: No known data errors
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 1.42G 21.7G 96K /
rpool/ROOT 1.32G 21.7G 96K none
rpool/ROOT/ubuntu 1.32G 21.7G 1.30G /
rpool/backup 152K 21.7G 96K /backup
rpool/home 888K 21.7G 108K /home
rpool/home/root 660K 21.7G 136K /root
rpool/srv 152K 21.7G 96K /srv
rpool/var 87.7M 21.7G 96K /var
rpool/var/cache 68.9M 21.7G 68.9M /var/cache
rpool/var/log 18.2M 21.7G 3.04M legacy
rpool/var/mail 152K 21.7G 96K /var/mail
rpool/var/spool 200K 21.7G 120K /var/spool
rpool/var/tmp 120K 21.7G 120K legacy
spool1 21.7G 393G 24K /spool1
spool1/zpool-cache-1 389M 393G 364M -
spool1/zpool-slog-1 21.3G 412G 2.57G -
spool2 21.7G 393G 24K /spool2
spool2/zpool-cache-2 389M 393G 364M -
spool2/zpool-slog-2 21.3G 412G 2.57G -
zpool 72.3G 8.71T 96K /zpool
zpool/test 72.0G 8.71T 13.8G /zpool/test
# time bonnie++ -d bonnie/ -s 1024G -n 0 -m TEST -f -b -u jhh
Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
TEST 1024G 1330380 80 577493 69 2061335 77 322.7 35
Latency 104ms 388ms 161ms 591ms
1.97,1.97,TEST,1,1532728953,1024G,,,,1330380,80,577493,69,,,2061335,77,322.7,35,,,,,,,,,,,,,,,,,,,104ms,388ms,,161ms,591ms,,,,,,
real 53m35.699s
user 1m39.283s
sys 37m22.875s
| Version 1.97 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| TEST | 1024G | 1330380 | 80 | 577493 | 69 | 2061335 | 77 | 322.7 | 35 | |||||||||||||||||
| Latency | 104ms | 388ms | 161ms | 591ms | Latency | |||||||||||||||||||||
# time bonnie++ -d bonnie/ -s 1024G -n 16 -m TEST -f -b -u jhh
Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
TEST 1024G 1284610 78 573617 68 2053432 77 333.5 37
Latency 101ms 499ms 138ms 1043ms
Version 1.97 ------Sequential Create------ --------Random Create--------
TEST -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 16017 49 +++++ +++ 11639 38 15523 49 +++++ +++ 15950 49
Latency 1878us 658us 309ms 1409us 23us 1258us
1.97,1.97,TEST,1,1532741832,1024G,,,,1284610,78,573617,68,,,2053432,77,333.5,37,16,,,,,16017,49,+++++,+++,11639,38,15523,49,+++++,+++,
15950,49,,101ms,499ms,,138ms,1043ms,1878us,658us,309ms,1409us,23us,1258us
real 54m23.006s
user 1m40.360s
sys 37m17.166s
| Version 1.97 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| TEST | 1024G | 1284610 | 78 | 573617 | 68 | 2053432 | 77 | 333.5 | 37 | 16 | 16017 | 49 | +++++ | +++ | 11639 | 38 | 15523 | 49 | +++++ | +++ | 15950 | 49 | ||||
| Latency | 101ms | 499ms | 138ms | 1043ms | Latency | 1878us | 658us | 309ms | 1409us | 23us | 1258us | |||||||||||||||
Otherwise same, but:
- zpool SLOG on NVME partition #5
- zpool CACHE on NVME partiton #6
Device Boot Start End Sectors Size Id Type
/dev/nvme0n1p4 101713920 1000215215 898501296 428.4G 5 Extended
/dev/nvme0n1p5 101715968 143659007 41943040 20G 83 Linux
/dev/nvme0n1p6 143661056 1000215215 856554160 408.4G 83 Linux
# zpool status zpool
pool: zpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500b0c19e30-part1 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500b0c1aad4-part1 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
/dev/disk/by-id/nvme-eui.002538c181b06801-part5 ONLINE 0 0 0
/dev/disk/by-id/nvme-eui.002538c181b06806-part5 ONLINE 0 0 0
cache
/dev/disk/by-id/nvme-eui.002538c181b06801-part6 ONLINE 0 0 0
/dev/disk/by-id/nvme-eui.002538c181b06806-part6 ONLINE 0 0 0
errors: No known data errors
# time bonnie++ -d bonnie/ -s 256G -n 16 -m TEST -f -b -u jhh
Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
TEST 256G 1443011 83 1226521 94 3544496 99 9670 682
Latency 101ms 103ms 344us 36726us
Version 1.97 ------Sequential Create------ --------Random Create--------
TEST -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 555 10 +++++ +++ 529 12 544 10 +++++ +++ 541 11
Latency 10900us 638us 10511us 14508us 21us 10643us
1.97,1.97,TEST,1,1532763127,256G,,,,1443011,83,1226521,94,,,3544496,99,9670,682,16,,,,,555,10,+++++,+++,529,12,544,10,+++++,+++,541,11,,101ms,103ms,,344us,36726us,10900us,638us,10511us,14508us,21us,10643us
real 10m9.211s
user 0m19.820s
sys 7m13.818s
| Version 1.97 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| TEST | 256G | 1443011 | 83 | 1226521 | 94 | 3544496 | 99 | 9670 | 682 | 16 | 555 | 10 | +++++ | +++ | 529 | 12 | 544 | 10 | +++++ | +++ | 541 | 11 | ||||
| Latency | 101ms | 103ms | 344us | 36726us | Latency | 10900us | 638us | 10511us | 14508us | 21us | 10643us | |||||||||||||||
Otherwise same, but zpool has no SLOG or CACHE.
# time bonnie++ -d bonnie/ -s 256G -n 16 -m TEST -f -b -u jhh Using uid:1000, gid:1000.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
TEST 256G 1386494 80 1183145 93 3587561 99 1882 160
Latency 104ms 100ms 823us 159ms
Version 1.97 ------Sequential Create------ --------Random Create--------
TEST -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 104 3 +++++ +++ 101 3 104 3 +++++ +++ 98 3
Latency 119ms 704us 115ms 101ms 19us 130ms
1.97,1.97,TEST,1,1532758208,256G,,,,1386494,80,1183145,93,,,3587561,99,1882,160,16,,,,,104,3,+++++,+++,101,3,104,3,+++++,+++,98,3,,104ms,100ms,,823us,159ms,119ms,704us,115ms,101ms,19us,130ms
real 19m6.612s
user 0m20.388s
sys 7m28.235s
| Version 1.97 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| TEST | 256G | 1386494 | 80 | 1183145 | 93 | 3587561 | 99 | 1882 | 160 | 16 | 104 | 3 | +++++ | +++ | 101 | 3 | 104 | 3 | +++++ | +++ | 98 | 3 | ||||
| Latency | 104ms | 100ms | 823us | 159ms | Latency | 119ms | 704us | 115ms | 101ms | 19us | 130ms | |||||||||||||||
| Version 1.97 | Sequential Output | Sequential Input | Random Seeks | Sequential Create | Random Create | |||||||||||||||||||||
| Size | Per Char | Block | Rewrite | Per Char | Block | Num Files | Create | Read | Delete | Create | Read | Delete | ||||||||||||||
| K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | K/sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | /sec | % CPU | |||
| ZVOL#1 | 1024G | 1330380 | 80 | 577493 | 69 | 2061335 | 77 | 322.7 | 35 | |||||||||||||||||
| Latency | 104ms | 388ms | 161ms | 591ms | Latency | |||||||||||||||||||||
| ZVOL#2 | 1024G | 1284610 | 78 | 573617 | 68 | 2053432 | 77 | 333.5 | 37 | 16 | 16017 | 49 | +++++ | +++ | 11639 | 38 | 15523 | 49 | +++++ | +++ | ||||||
| Latency | Latency | |||||||||||||||||||||||||
| PARTITIONS | 256G | 1443011 | 83 | 1226521 | 94 | 3544496 | 99 | 9670 | 682 | 16 | 555 | 10 | +++++ | +++ | 529 | 12 | 544 | 10 | +++++ | +++ | 541 | 11 | ||||
| Latency | 101ms | 103ms | 344us | 36726us | Latency | 10900us | 638us | 10511us | 14508us | 21us | 10643us | |||||||||||||||
| WITHOUT | 256G | 1386494 | 80 | 1183145 | 93 | 3587561 | 99 | 1882 | 160 | 16 | 104 | 3 | +++++ | +++ | 101 | 3 | 104 | 3 | +++++ | +++ | 98 | 3 | ||||
| Latency | 104ms | 100ms | 823us | 159ms | Latency | 119ms | 704us | 115ms | 101ms | 19us | 130ms | |||||||||||||||