readme.md

How to monitor Resque with God and do it all with Capistrano on Ubuntu

Published on Rendered Text blog. Credit to thbar's original gist - GitHub prevented me of just forking it with 404s.

First, install god gem:

sudo gem install god

Create a master configuration file and prepare it for loading the instructions for Resque:

sudo mkdir /etc/god
sudo vim /etc/god/master.conf

Make god a service that runs on boot:

sudo cp god-service /etc/init.d
sudo chmod +x /etc/init.d/god-service
sudo update-rc.d god-service defaults

At this point you're able to do /etc/init.d/god-service start|stop|status.

Prepare your app's resque-#{environment}.god (I haven't figured out how to have one file handle multiple environments).

Add the deploy task to your Capistrano file. It assumes that your deploy user has passwordless sudo permissions. To do that, run visudo and enter

%deployuser         ALL = (ALL) NOPASSWD: ALL

At this point cap deploy should Just Work™.

deploy.rb

namespace :deploy do
  desc "Hot-reload God configuration for the Resque worker"
  task :reload_god_config do
    sudo "god stop resque"
    sudo "god load #{File.join(deploy_to, 'current', 'config', 'resque-' + rails_env + '.god')}"
    sudo "god start resque"
  end
end

# append to the bottom:

after :deploy, "deploy:reload_god_config"

god-service

#!/bin/bash
#
# god       Startup script for god (http://god.rubyforge.org)
#
# chkconfig: - 85 15
# description: God is an easy to configure, easy to extend monitoring \
#              framework written in Ruby.
#

CONF_DIR=/etc/god
GOD_BIN=/usr/local/bin/god
RUBY_BIN=/usr/bin/ruby
RETVAL=0

# Go no further if config directory is missing.
[ -d "$CONF_DIR" ] || exit 0

case "$1" in
    start)
      # Create pid directory
      $RUBY_BIN $GOD_BIN -c $CONF_DIR/master.god
      RETVAL=$?
  ;;
    stop)
      $RUBY_BIN $GOD_BIN terminate
      RETVAL=$?
  ;;
    restart)
      $RUBY_BIN $GOD_BIN terminate
      $RUBY_BIN $GOD_BIN -c $CONF_DIR/master.god
      RETVAL=$?
  ;;
    status)
      $RUBY_BIN $GOD_BIN status
      RETVAL=$?
  ;;
    *)
      echo "Usage: god {start|stop|restart|status}"
      exit 1
  ;;
esac

exit $RETVAL

master.god

# load scalr environment variables
God.load "/etc/god/scalr.god"

# load in all production config files
ENV['PRODUCTION'] = 'true'
God.load "/var/www/conf/*.god"
God.load "/var/www/*/conf/*.god"

# remove production env variable
ENV['PRODUCTION'] = 'false'
ENV::delete('PRODUCTION')

# load in all development config files
ENV['DEVELOPMENT'] = 'true'
God.load "/code/*/conf/*.god"

resque-production.rb

God.watch do |w|
  w.name = 'resque'
  w.interval = 30.seconds
  w.env = { 'RAILS_ENV' => 'production', 'QUEUE' => '*' }
  w.uid = 'deployuser'
  w.gid = 'deployuser'
  w.dir = File.expand_path(File.join(File.dirname(__FILE__),'..'))
  w.start = "bundle exec rake jobs:work"
  w.start_grace = 10.seconds
  w.log = File.expand_path(File.join(File.dirname(__FILE__), '..','log','resque-worker.log'))

  # restart if memory gets too high
  w.transition(:up, :restart) do |on|
    on.condition(:memory_usage) do |c|
      c.above = 200.megabytes
      c.times = 2
    end
  end

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
      c.interval = 5.seconds
    end

    # failsafe
    on.condition(:tries) do |c|
      c.times = 5
      c.transition = :start
      c.interval = 5.seconds
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_running) do |c|
      c.running = false
    end
  end
end

resque.rake

require 'resque/tasks'

task "resque:setup" => :environment do
  ENV["QUEUES"] = "critical,high,low"
end

desc "Alias for resque:work (to let us imagine that we are on Heroku)"
task "jobs:work" => "resque:work"