Hutch deployment notes

hutch-deployment-notes.txt

When you install the Hutch gem, an executable called `hutch` will be installed.
Assuming you're using bundler, you can run Hutch by executing the hutch
executable. If it's executed in the working directory of a Rails app, the Rails
app will be auto-detected and any consumers in the `app/consumers` directory
will be auto loaded. If you're not using Rails, or have your consumers in
non-standard locations, pass `--require` flags to the hutch executable to
specify your consumer paths.

To run the Hutch executable, it's probably worth using a process supervisor
such as upstart, supervisord, god, etc., just as you'd use with your app
server. It should be fairly stable, but you'll probably want to restart it in
the event that the process dies.

In terms of monitoring, we currently monitor the RabbitMQ queues that Hutch
creates. You want to make sure that these queues are draining, or at least
don't grow forever. This could be implemented as a check that runs every
minute, and says "make sure no queue has > 1k items in 5 times in a row", or
you could just set a hard threshold so you get an alert when your queues hit
10k messages. We opt for the former as we have large spikes of messages every
day, which is considered usual behavior.