For a long time I've been really impacted by the ease of use Cassandra and CockroachDB bring to operating a data store at scale. While these systems have very different tradeoffs what they have in common is how easy it is to deploy and operate a cluster. I have experience with them with cluster sizes in the dozens, hundreds, or even thousands of nodes and in comparison to some other clustered technologies they get you far pretty fast. They have sane defaults that provide scale and high availability to people that wouldn't always understand how to achieve it with more complex systems. People can get pretty far before they have to become experts. When you start needing more extreme usage you will need to become an expert of the system just like any other piece of infrastructure. But what I really love about these systems is it makes geo-aware data placement, GDPR concerns potentially simplified and data replication and movement a breeze most of the time.
Several years ago the great Andy Gross did a talk challenging distributed and database system designers to think like the Unix philosophy of composable pieces. This was way before serverless or when it was popular to separate compute from storage. It made me think about so many different system ideas.
In this document I propose a system I am naming Recoil. It's a cluster fuck combination of ideas in my head about data and distributed applications that I want to explore. In some ways it feels like deploying WAR files to a Tomcat cluster. Yup, you're gonna hate me in this document.
Recoil is my next fun project. It's been a few years since I experimented with the Haywire project and pushed my knowledge about computer performance and I want to build something fun and exciting again.
Recoil is a composable WASM powered database kit that can be used as a framework to build distributed applications or run as a clustered data store.
Recoil is almost like Kubernetes in the sense it is a set of building blocks that you can host functionality within. Recoil will provide the following foundation:
- Easy cluster membership and easy shard/partition management like Cassandra and CockroachDB.
- Transactions using multi-raft consensus groups assigned to each shard/partition replica set.
- Storage and Consensus API's exposed as WASM API's.
- Geo-aware operations like Cassandra and CockroachDB.
- Ability to dynamically move hot partitions like CockroachDB.
That's it. I know, confusing right? If you want Recoil to actually do something then you need to deploy packages on the Recoil cluster that enables the functionality you need.
Recoil is a clustered distributed hash table where multiple nodes host partitions of the overall data space with a user defined replication level. When you install a package on the Recoil cluster you are deploying a WASM artifact that gets deployed to all the nodes. Or we could do something fancy like node selection in Kubernetes. Now Recoil becomes a data processing grid and you deploy WASM code to nodes in the grid. That hurts my brain.
I guess Kubernetes takes a very process oriented approach to scheduling where Recoil takes a data distribution oriented appraoch to scheduling. This is probably a bad idea, but what the fuck I'm stubborn.
Starting the cluster could be something like this.
$ ./recoil_simulator --number-of-nodes=3
Recoil node f937c37e bootstrapped cluster hosting 265 partitions
Recoil node 949d9efa joined cluster hosting 265 partitions
Recoil node 20d2958a joined cluster hosting 265 partitions
$ recoilctl install redis-server
Something like recoilctl install redis would install a Redis protocol listener that someone wrote in Rust, or Javascript, or Go or C# or Java that is a Recoil plugin compiled into WASM. Or maybe you want to host S3 data on recoil recoilctl install s3 That means you can write a data model like a key/value store, a message queue or a redis cache and the network protocol listener in any programming language since Recoil will host the WASM code in the cluster.
Taking this idea to a wild idea could be that Recoil is a grid of data and WASM processing. It could partition the cluster itself to host different data and different WASM modules in different areas of the grid. This breaks my brain at the moment because it starts to feel like Kubernetes. In a way this idea really excites me though. At this point we need a Scheduler which could also be WASM provided so that custom scheduling could be written in any programming language.
At this point nothing stops people from implementing their HTTP API directly on the Recoil cluster with business rules written in any programming language they want. This sounds like the worst idea ever but sounds really fun haha.
Cluster membership and partition management using Dragonboat multi-group Raft consensus library. This seems like a pretty mature and high performance multi-group Raft library which is hard to find.
DHT algorithm based on Google's Consistent Hashing With Bounded Loads. This has helped companies like Vimeo get better utilization and performance out of one of their systems.
Key/Value store API's provided by Pebble or BBolt.
Wasmer WASM runtime to load and execute WASM packages across the Recoil cluster.
Orchestrating the distribution of WASM packages across a cluster of nodes seems really far away. I think the first prototype needs to just assume the WASM binaries are already present or else I'm going to spend all my time working on shipping binaries around.
I'm by no means trying to write a production quality code base, or any code at all in this experiment at the time of beginning this document. My main purpose is to experiment as far as I can and I don't know where it's going to go :) So far this is an idea I'm sharing and I am TOTALLY open to ideas (or even contributions if anyone gives a shit). Please share your thoughts!
I wanted to take this opportunity to learn Rust here but I need to be realistic. I might struggle to push this idea forward period so I need to reduce my challenges during this experimentation. Go will be good enough for now.