Proposal on #cassandra for changing up the relay

Further notes

The "relay" is actually a chameleon - it can be either a straight rules-based relay 
(send A -> B, send B->z, etc. based on a regular expression match on the stat name), 
it can be a consistent hashing relay which spreads load across a ring, or it can be 
an aggregated consistent hashing relay, which I really don't know anything about but 
will learn more about.

The only thing I'm really thinking about right now is the rules-based relay, but the 
same approach could be used for the others if that's needed.

It also leads to a very simple hack to deliver to e.g. an alternative backend.  E.g.
if we wanted to experiment with a cassandra backend, it would be easy to bypass the 
carbon-cache and have a small writer that just writes directly to cassandra.

gistfile1.txt

0:28:48            pcn | Heya #graphite, I've got some thoughts about the relay              │ ankesh11
10:29:13            pcn | Heya #graphite, I've got an issue with the relay and with twisted   │ antonw_
10:29:55            pcn | problem: the relay uses an interface to send metrics as pickled     │ apejens
                        | strings.  That interface in twisted will in turn write the string   │ athoune
                        | to a list, and underneath the relay it will buffer sending          │ azbarcea
10:30:44            pcn | When the distance between relays sending <-> receiving grows long   │ banjiewen
                        | enough, there appears to be a limit on how much data can be sent    │ BeanDip_
                        | from a single thread                                                │ behrendsj
10:32:00            pcn | I'm thinking that a better way to handle this would be to either    │ bensix2
                        | change from being a streaming producer, and allow twisted to feed   │ bfritz
                        | back flow control info, but I feel like the chunks it's sending     │ BiGBOi
                        | data in are too small to keep                                       │ bitprophet
10:32:01            pcn | up                                                                  │ bodgit_
10:33:24            pcn | Alternatively, I'm thinking about offloading the sending to         │ brdude
                        | subprocesses - queue stats out to disk and fork+exec to launch a    │ Bucciarati
                        | separate, smaller program, to send stats in parallel so as to save  │ cdeszaq
                        | on resources.                                                       │ Champi
10:35:43            pcn | By making the the interval between launching a worker a tunable     │ charlieS
                        | parameter, the number of workers a parameter, and the number of     │ chudler
                        | stats per pickle a parameter for the workers, it seems like it'd    │ Comradin
                        | provde for a way to scale horizontally on a system                  │ cparedes    ++

pros to this approach

- stats can be flushed to disk and manually replayed out to the remote relay even if the system is having problems.
- The sending is isolated in a simpler code path
- Queuing can be spooled to disk and so a much larger queue can be permitted