liveness.rake

desc 'Determine if Resque is Alive or not'
task :liveness, [:queue_name] => [:environment] do |_task, args|
  Resque.workers.select { |w| w.hostname == Socket.gethostname }.each do |worker|
    begin
      Process.kill(0, worker.pid)
    rescue Errno::ESRCH
      puts "Dead: Resque process with pid #{worker.pid} should exist, but cannot be found."
      exit 1
    end

    run_at = worker.job["run_at"]

    if run_at
      now = DateTime.now.strftime("%s").to_i
      run_at = DateTime.parse(run_at).strftime("%s").to_i
      diff = (now - run_at)
      if diff > 300
        puts "Dead: A worker job has been running for #{diff} seconds"
        exit 1
      end
    end
  end
  puts "Alive"
end

How about making a class out this task so we can write test around them (mostly for documenting how we are figuring out 'liveness')?

Some more ideas on deducing Resque worker states:
https://github.com/resque/resque-web/blob/master/app/views/resque_web/workers/show.html.erb

Yeah @devdude we were chatting about that on hipchat, something like this?
https://gist.github.com/bloodycelt/28d691e2c4d8e9d41eb7aabcc109f088