Elevator plan

elevator.md

First Article (Adressing the problem)

Special needs

Here at Botify

• Server logs data analytics storage (TeraBytes order)

• Bulk write/read GigaBytes datas loads that would not suit in server's memory

• Need for persistence

• LevelDB showed amazing performances suiting our needs : Benchmarks

• But levelDB is an embedded db engine, we needed something able to answer the phone (Picture : "Yes, this is dog speaking")

  • Needed to be able to access a LevelDB backend from multiple processes
  • Example : adressing the sequential url ids problem

• A leveldb-server already existed, but was more of a script rather than a real server. Plus, we are perfectionists.

So was born Elevator

• High-performance on-disk key-value store.
• Suitable for bulk operations, without exploding RAM consumption
• Asynchronous, multi-databases, mutli-clients
• Batch system

---

Second article (Under the hood)

How it is designed

• Python : Why not using low-level language
• Leveldb - plyvel : why we left py-leveldb
• ZeroMQ - Msgpack

Benchmarks

Future

• commands pipeline
• multiple backends?
• Restful api
• supervision dashboard