LevelDb notes

LevelDb-notes.md

LevelDb

Features

• embedded
• light weight
• thread safe
• fast
• good compression
• arbitrary byte arrays
• sorted by keys
• snapshots
• WriteBatch (Put and/or Del are atomic)
• BloomFilter
• Forward and backward iterator

Like Redis?

• Bit like redis but lacks data types like lists and sets
• More reliable read/writes than Redis and we don't store our entire data in memory

Usages

• Good for cases where we don't want to worry about RAM but need speed very close to memory access.
  • Smartphones, Raspberry Pi, etc
  • Offline web apps

Writes

• Writes use a similar policy as Cassandra and BigTable inspired dbs
  • Cassandra: CommitLog -> Memtable -> full? -> SSTable
  • LevelDb: Log -> SST

Reads

• Reads merge the log and table files and use cache on top of them.

Sorted table files (SST)

• ~2Mb each
• 4K blocks
• Final block is index
• BloomFilter for lookups

Table file Hierarchy

• Log: Reached 4MB? flush to set of L0 SST
• L0: Got 4 SST files? compact to single L1 file
• L1: 10MB? compact into L2
• L2: 100MB? compact into L3
• L3+: 10x previous level? compact into next level

Data Safety

• Only case of data getting lost is when OS crashes
• Could use: {sync: true} which would fire the callback only once data is written to disk, would make writes really slow though.
• Use level-hook package to direct writes to multiple locations of add redundancy on top of levelDb.

Backup

Copying Directory

• Close db, copu whole directory
• If closing db is not possible, a running copy should be ok but might hit middle of a compaction.
  • Run:

levelup.repair(location, callback)

if that happens

Streaming backup data

• Open another db and stream the source into target

function copy (srcdb, dstdb, callback) {
  srcdb.createReadStream().pipe(dstdb.createWriteStream()).on('close', callback)
}

• createWriteStream() has been removed. Check this instead.

API

Example

var levelup = require('levelup')

// 1) Create our database, supply location and options.
//    This will create or open the underlying LevelDB store.
var db = levelup('./mydb')

// 2) put a key & value
db.put('name', 'LevelUP', function (err) {
  if (err) return console.log('Ooops!', err) // some kind of I/O error

  // 3) fetch by key
  db.get('name', function (err, value) {
    if (err) return console.log('Ooops!', err) // likely the key was not found

    // ta da!
    console.log('name=' + value)
  })
})

Common Operations

• Complete guide
• open()
• close()
• put()
• get()
• del()
• batch()
• isOpen()
• isClose()
• createReadStream()
• createKeyStream()
• createValueStream()

LevelUP

• A fancy wrapper around LevelDown (C++ interface between Node and LevelDb) to allow
  • optional args
  • deferred-till-open
  • streams
  • JSON and other encoding types

Further Reading

• Examples in C++