gistfile1.txt

17:58 < tmm1> kevinclark: so updating max_per_second did it?
18:05 < kevinclark> the last of many tweaks :) previously I think I had things poorly configured (didn't grow well from 1 box to 5). So now we've got one machine doing relay/agg, 5 doing storage/render. 
                    memcached on each storage box (which only hits locally)
18:05 < kevinclark> there was the config issue where each webserver was hitting each carbon
18:05 < kevinclark> now it only hits its own
18:05 < kevinclark> I had to bump up queue length on agg and relay
18:05 < kevinclark> and I had to bump up message size across the board
18:06 < kevinclark> now it's reasonable. not great, but good enough I think
18:06 < kevinclark> 4-8 second render uncached
18:06 < kevinclark> sometimes faster or slower, but usually about there
18:07 < kevinclark> before it was 10-20
18:07 < kevinclark> for a single datapoint (now we're doing up to 50 in 4-8)
18:10 < tmm1> nice
18:24 < kevinclark> gonna blog it / tweak some documentation sometime in the next week I hope
18:35 < tmm1> awesome