Process multi-jobs per fork of resque

resque.rb

Resque.after_fork do |job|
  jobs_performed = 0
  kill_fork_at = Time.now.to_i + (ENV['MINUTES_PER_FORK'].to_i * 60)
  worker = job.worker
  first_job = job

  worker.procline "Processing #{job.queue} since #{Time.now.to_i} until #{kill_fork_at}"

  first_job.perform
  # rely on parent error handling
  worker.log "done: #{first_job.inspect}"
  worker.done_working
  jobs_performed += 1
  
  while Time.now.to_i < kill_fork_at && !worker.shutdown?
    if job = worker.reserve
      worker.working_on(job)
      job.perform
      # rely on parent error handling
      worker.log "done: #{job.inspect}"
      worker.done_working
      jobs_performed += 1                
    else
      sleep(1)
    end
  end
  first_job.instance_eval("def perform; end")

end

The instance_eval is to prevent the first_job from running again when this after_fork finishes.

If you look at the source of resque, the after_fork hook is called after_forking and before running the job. This after_fork hook runs the first job first, to maintain the correct job order.

Thanks Adam, I hadn't really grokked too much of the callback code in Resque, but this makes sense. Your patch is working great for us in production, thanks!

Glad to hear. We are using it as well in production and haven't had any issues.

We recently removed this patch from our setup because it was causing the Failure backend to report incorrect failures. When a job would fail, it would sometimes have arguments from a previous job / or the class name was mismatched.

mguterl have you been able to work-around the "incorrect failure reporting" thing?

We're now using resque-multi-job-forks

Thanks mguterl !

i used resque-multi-job-forks, but the workers quit when all the jobs are completed.
$ MINUTES_PER_FORK=1 COUNT=2 QUEUE=db rake resque:workers
(in /home/my_app)
(in /home/my_app)
The signal QUIT is in use by the JVM and will not work correctly on this platform
The signal USR1 is in use by the JVM and will not work correctly on this platform
The signal QUIT is in use by the JVM and will not work correctly on this platform
The signal USR1 is in use by the JVM and will not work correctly on this platform
155 were processed in this fork
245 were processed in this fork

$ _

Do you know why could that be?