Tests to do:
Create an environment with 4 (at least >2) servers each running a single process with 40 writing threads. Each writing thread should insert a new document at some prescribed rate (maybe make the threads and the document insertion a configuration parameter?)
Have each of these servers writing to one or more of the nosql "servers" (depending on architecture) -- maybe run one test where all the writes are going to one server, and another where they are spread.
Then lets see how long it takes for each of the servers in a cluster to "catch up" (e.g. what is the lag time average and std dev -- if thats even possible to compute) before a new document is replicated around.
Once we have a feel for this, lets repeat the test but pull one server out of the replication environment. We should do this test twice, once where we remove the server gracefully and once violently. In both cases we'll leave the server "offline" for several tens-of-minutes or hours. Then lets compute the number of documents the server is now "behind" by. Placing the server back in the replication loop and see how long it takes to catch up.
This will tell us in what manner we can take nodes down for maintenance or to use them as backup snapshot servers.
A customer might have: 15,000 devices each delivering logs every 5 minutes in which each delivery will contain between 5 and ten records. We can experiment with different simultaneous device counts spacing of deliveries.