What does the ultimate database look like? Maybe the ultimate database of the future going to be a many-headed hydra that will attempt to solve all problems? Maybe marketing teams are going to be at the heart of its success? Perhaps it will grow tentacles and become violent toward its creator?
Unless you're a contributor to the software, your probably limited to understanding the value propositions and a subset of features. Maybe your an expert. Probably not. This corner is usually dark. A database has historically been a black box. You pick one that seems like the best fit and trust it.
Maybe the database of the future should just be a library, more like BerkeleyDB, I really don't want an entire server. Someone could just add a server on top of it if they really wanted one. It would also be nice if understanding it (in its entirety) wouldn't represent a big investment.
The complexity of a system can be quantified as the number of steps that it takes to bring it to its intended state. My general rule with complexity is that proof must be provided that something is needed, otherwise it doesn't belong.
My ideal database would start with only the absolute minimum viable means to efficiently store and retrieve arbitrary data. All additional features would be abstracted into discrete modules. This model was proven to be quite successful with Node.js. Load balancing, Map Reduce, Replication, etc. could all be modules!
Of course not. That's crazy. Someone will implement a SQL module.
No. The future is never now. But LevelDB aligns more with what I'm looking for in the "ultimate database of the future".
Leveldb is a small C++ library. It's classified as a key value store with a Log-Structured Merge-Tree (LSM) architecture. It's performs very fast range queries.
The contents of the database are stored in a set of files in the filesystem. You can learn about all of the files that get created here. I'll cover the ones that are relevant to getting a basic understanding of leveldb.
Log files (*.log) are append-only. They contain a sequence of updates. A copy of the log file lives in memory. Writes get logged to this structure and reads happen here first so that recent updates are reflected. This memory structure is periodically flushed to disk. Keeping a percentage of the recently active data as well as indexes in memory makes leveldb efficient for both random reads, writes and range queries. The search performance is O(log N) with a very large branching factor.
Leveldb persists data in Sorted String Tables (*.sst). SSTs are files filled with immutable, arbitrary, key-value pairs sorted on their keys. Keys and values are arbitrary blobs. Each entry's value is either a value for the key, or a deletion marker for the key. The indexes for these files are also loaded into memory so random writes are also fast. SSTs can be used to exchange up to Terabytes of sorted data segments.
SSTs are organized into a sequence of levels and continuously compacted over time.
"It empowers you to write your own db, from a super light but fast filesystem abstraction to a beefed up server with custom replication schemes and other application-logic right at the heart of this super fast thing.[…]" -- @juliangruber
"You could implement something like lambda architecture completely on top of leveldb. It's a basic building block, you can build your own db abstraction with your own trade offs on top of it." -- @raynos
"LevelDB makes data persistence easy. Instead of storing your data structures in memory, store them in a LevelDB database and fetch when needed -- this way you can store a ton of data and not have to worry about RAM and you get to keep that data across restarts. This is how I mainly use it, LevelUP makes it super simple and fetching & processing large amounts via a readStream() is just so nice to work with. If you use the inbuilt JSON encoding then you get to pretend the data is in memory (except where serialization/deserialization may change the form of your data, like Date objects), since LevelDB is so fast and all operations are async the speed impact of storing on disk is hardly noticeable." -- @rvagg
Levelup is a driver for LevelDB and people have started writing lots of useful modules, plugins and tools surrounding it.
To get my hands dirty, I wrote a command line tool and REPL (with autosuggestion and autocomplete for keys) to help query and manage leveldb instances. Star it and tweet about it if you like it.