In-Memory DB tests, part 1

In-Memory DB Tests

Inspired by this RocksDB post, I started playing with this "readwhilewriting" test as well. 
https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks

Of course the data here is only a tiny fraction of the size of the RocksDB report. (I have
a larger test running that's of a more comparable size, but it will probably be a couple
more days before all of the tests finish.)

So - tested LevelDB 1.17, Basho LevelDB 1.9, BerkeleyDB 5.3.21, HyperLevelDB 1.14,
LMDB 0.9.12rc, RocksDB 3.2, TokuDB 4.6.119, and WiredTiger 2.2.0 (both LSM and Btree).
That's a lot of tests to run, trust me.

The test system is an Asus NP56D laptop with quadcore AMD A10-4600M APU and 16GB DRAM. The
test first loads a database with 10M small records (16 byte keys, 100 byte values) as fast
as possible, then fires off 4 threads to read the DB and 1 thread to write it, all using
random keys. The writer is constrained to no more than 10240 writes per second. The
purpose of the test is to measure read throughput.

Summary...

Engine            Load Time           Load Size    Writes/Sec   Reads/Sec        Run Time           Final Size
             Wall,   User,   Sys         KB                                 Wall,   User,   Sys         KB
LevelDB      00:17s  00:22s  00:03s     1123024      10231       158919     04:12s  15:00s  01:15s    1186640
Basho        00:19s  00:35s  00:07s     2536940      10231        61490     10:51s  40:36s  02:14s    1450180
BerkeleyDB   01:07s  01:01s  00:06s     1918364       9584        77603     08:36s  23:30s  09:51s    1940912
HyperLvlDB   00:19s  00:24s  00:03s     1123224      10196       146328     04:34s  16:37s  01:01s    1146308
LMDB         00:05s  00:04s  00:01s     1258128      10227      1501891     00:27s  01:44s  00:01s    1289908
RocksDB      00:11s  00:17s  00:03s     1130692      10232        82226     08:08s  29:33s  02:28s    1514476
TokuDB       01:03s  00:55s  00:33s     1612296       9801       114568     05:54s  17:59s  03:23s    2326852
WiredLSM     00:34s  01:14s  00:09s     2044484      10214       280009     02:23s  09:04s  00:06s    1407764
WiredBtree   00:09s  00:08s  00:01s     1190944      10207       128953     05:11s  14:55s  05:01s    2382104


The same test, using 20M records instead of 10M
Engine            Load Time           Load Size    Writes/Sec   Reads/Sec        Run Time           Final Size
             Wall,   User,   Sys         KB                                 Wall,   User,   Sys         KB
LevelDB      00:35s  00:45s  00:07s     2246004      10232        24626     54:09s  96:33s  58:33s    3422060
Basho        00:41s  01:23s  00:17s     6281124      10233        47742     27:56s 103:29s  06:47s    2614212
BerkeleyDB   02:08s  01:57s  00:11s     3262628       9475        74154     17:59s  49:08s  20:47s    3258468
HyperLvlDB   00:38s  00:49s  00:06s     2247448      10192       135028     09:53s  36:05s  02:10s    2291036
LMDB         00:11s  00:08s  00:02s     2516192      10241      1444797     00:56s  03:35s  00:02s    2547976
RocksDB      00:22s  00:35s  00:06s     2256032      10232        64513     20:40s  74:14s  07:22s    3348364
TokuDB       01:45s  01:41s  00:47s     2718360       9884       109104     12:16s  37:57s  07:10s    3898032
WiredLSM     01:09s  02:37s  00:19s     2405856      10219       255603     05:14s  20:12s  00:11s    2762792
WiredBtree   00:17s  00:15s  00:02s     2381876      10218       105235     12:41s  36:47s  12:45s    4763968

The binaries and script to run this test are at http://highlandsun.com/hyc/bench-20140609.tgz
Raw data for the 10M record test is at http://highlandsun.com/hyc/raw10M.txt

Software Versions

Software revisions used:

violino:/home/software/leveldb> g++ --version
g++ (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


violino:/home/software/leveldb> git log -1 --pretty=format:"%H %ci" master
e353fbc7ea81f12a5694991b708f8f45343594b1 2014-05-01 13:44:03 -0700

violino:/home/software/basho_leveldb> git log -1 --pretty=format:"%H %ci" develop
16b22c8198975b62a938dff9910f4432772d253a 2014-06-06 12:25:40 -0400

violino:/home/software/db-5.3.21> ls -l README 
-rw-r--r-- 1 hyc hyc 234 May 11  2012 README

violino:/home/software/HyperLevelDB> git log -1 --pretty=format:"%H %ci" releases/1.0
a7a707e303ec1953d08cbc586312ac7b2988eebb 2014-02-10 09:43:03 -0500

violino:~/OD/mdb> git log -1 --pretty=format:"%H %ci" 
a93810cc3d1a062bf5edbe9c14795d0360cda8a4 2014-05-30 23:39:44 -0700

violino:/home/software/rocksdb> git log -1 --pretty=format:"%H %ci"
0365eaf12e9e896ea5902fb3bf3db5e6da275d2e 2014-06-06 18:27:44 -0700

violino:/home/software/sophia> git log -1 --pretty=format:"%H %ci"
3f16d0ed3f512dfa5d4d2036bba7deb4fe6ae4ec 2014-02-25 20:56:53 +0400

violino:/home/software/sqlite3.git> git log -1 --pretty=format:"%H %ci" master
c18a797d6853081493732fa9a3a77e1334a9ef12 2013-09-12 19:28:34 -0700

violino:/home/software/ft-index> git log -1 --pretty=format:"%H %ci" master
f51c7180db1eafdd9e6efb915c396d894c2d0ab1 2014-05-30 12:58:28 -0400

violino:/home/software/kyotocabinet-1.2.76> ls -l ChangeLog 
-rw-r--r-- 1 hyc hyc 29158 May 24  2012 ChangeLog

violino:/home/software/wiredtiger> git log -1 --pretty=format:"%H %ci"
91da74e5946c409b8e05c53927a7b447129a6933 2014-05-21 17:05:08 +1000

Hi Howard, do you have a build script for the dependencies?

Also, what's the goal with this work -- what's driving the choice of data volume, concurrency, test machine, etc.? Do they represent some real-world workload?

Hi Michael, I just followed whatever build instructions for each of those other libraries, nothing special. However in every case I disabled use of any 3rd party libraries (e.g. tcmalloc or jemalloc) and disabled use of any compression libraries.

I actually don't have a use case for this myself, was just poking at the RocksDB performance report linked above. I can't think of any sane reason to run a production DB on a tmpfs. It's a horrible waste of RAM, and a horrible waste of time to work with all that data on a non-persistent medium.

By the way, the full report is now online http://symas.com/mdb/inmem/

As an afterthought, the readwhilewriting test is probably close to what an LDAP workload looks like - very read-heavy.

As another afterthought, I would consider using NVDIMMs and a reset-survivable RAMdisk, like we used to do back in the 1980s... https://groups.google.com/d/msg/comp.sys.atari.st/4Vg8rGUuGC4/TJrFrnt5WaMJ