Tests to do:
Create an environment with 4 (at least >2) servers each running a single process with 40 writing threads. Each writing thread should insert a new document at some prescribed rate (maybe make the threads and the document insertion a configuration parameter?)
Have each of these servers writing to one or more of the nosql "servers" (depending on architecture) -- maybe run one test where all the writes are going to one server, and another where they are spread.
Then lets see how long it takes for each of the servers in a cluster to "catch up" (e.g. what is the lag time average and std dev -- if thats even possible to compute) before a new document is replicated around.
Once we have a feel for this, lets repeat the test but pull one server out of the replication environment. We should do this test twice, once where we remove the server gracefully and once violently. In both cases we'll leave the server "offline" for several tens-of-minutes or hours. Then lets compute the number of documents th