[note: this was originally published on 08/27/2012. Other writing can be found here.]
Monitoring tooling has been a fascination of mine since I began hacking on computing systems. Every ops engineer wants to write their own. I get that. I'm one of those guys, too.
However, these endeavours always end up spitting out large, monolithic applications that just aren't as good as the already existing (and somewhat crappy) monolithic applications. I'd prefer to avoid that approach, if only to make it a point that the monitoring story can never be finalized. We will never reach an end state, but the more flexible our tooling the better.
Traditional Unix philsophy gives us something to go on here. What if, instead of writing large applications, we just wrote a bunch of tiny tools. AND - rather than writing each of these tiny tools as sexy network daemons (which