New Sentry Buffers

gistfile1.txt

When an update comes into the system we fire off something like the following:

>>> app.buffer.incr(
>>> 	Group,  # model class,
>>> 	{'times_seen': 1},  # counters
>>> 	{'id': group.id},  # filter restrictions, sometimes a composite key
>>> 	{'last_seen': now},  # metadata to update when buffer is processed
>>> )

This would get stored in a hash key as following:

>>> {
>>> 	_filter => '{"id": "1"}',
>>> 	_model => 'sentry.models.Group',
>>> 	times_seen => 5
>>> 	last_seen => unix timestamp,
>>> }

Future updates would always overwrite the values in extra (e.g. last_seen) and
do hincr's on the counters. The filters would be serialized as a json value,
and the model would just be it's class path.

We would also append the full hash key name to a pending key. The pending key
would be a simple set, and it would map to a random node in the Redis cluster.
This means effectively that each Redis server would have a 'pending' key that
has a random (deterministic) set of pending updates that could be flushed
indepdently.


Periodically (likely every 10 seconds) a task if fired which flushes the pending
key on each node, and fires off a subsequent job. This would be something like:

>>> multi
>>> smembers pending
>>> del pending
>>> exec

Each individual job, when run, would start with:

>>> multi
>>> hget key
>>> srem pending key
>>> exec

This would ensure that we process as much of the buffer at one time as possible.

It would take the model and filters and do a simple create or update clause given
the data.