Process multi-jobs per fork of resque

resque.rb

Resque.after_fork do |job|
  jobs_performed = 0
  kill_fork_at = Time.now.to_i + (ENV['MINUTES_PER_FORK'].to_i * 60)
  worker = job.worker
  first_job = job

  worker.procline "Processing #{job.queue} since #{Time.now.to_i} until #{kill_fork_at}"

  first_job.perform
  # rely on parent error handling
  worker.log "done: #{first_job.inspect}"
  worker.done_working
  jobs_performed += 1
  
  while Time.now.to_i < kill_fork_at && !worker.shutdown?
    if job = worker.reserve
      worker.working_on(job)
      job.perform
      # rely on parent error handling
      worker.log "done: #{job.inspect}"
      worker.done_working
      jobs_performed += 1                
    else
      sleep(1)
    end
  end
  first_job.instance_eval("def perform; end")

end

Glad to hear. We are using it as well in production and haven't had any issues.

We recently removed this patch from our setup because it was causing the Failure backend to report incorrect failures. When a job would fail, it would sometimes have arguments from a previous job / or the class name was mismatched.

mguterl have you been able to work-around the "incorrect failure reporting" thing?

We're now using resque-multi-job-forks

Thanks mguterl !

i used resque-multi-job-forks, but the workers quit when all the jobs are completed.
$ MINUTES_PER_FORK=1 COUNT=2 QUEUE=db rake resque:workers
(in /home/my_app)
(in /home/my_app)
The signal QUIT is in use by the JVM and will not work correctly on this platform
The signal USR1 is in use by the JVM and will not work correctly on this platform
The signal QUIT is in use by the JVM and will not work correctly on this platform
The signal USR1 is in use by the JVM and will not work correctly on this platform
155 were processed in this fork
245 were processed in this fork

$ _

Do you know why could that be?