Set up background workers on Heroku with Reqsque/Redis

rails-background-workers-with-resque.md

Set Up Background Workers with Resque/Redis on Heroku

LOCAL SETUP

Ensure Redis is installed on local machine

To install, run (assumes HomeBrew is already set up):
$ brew install redis

Start Redis server
$ redis-server

Run the following commands to confirm:

$ redis-cli ping  
PONG  
$ redis-cli  
redis 127.0.0.1:6379> set foo bar  
OK  
redis 127.0.0.1:6379> get foo  
"bar"  

Require the Resque gem in Gemfile

gem 'resque', "~> 1.22.0"

Run bundle install and bundle update to update gems and Gemfile.lock.

Run heroku run bundle install to install dependencies on Heroku.

Set application queue adapter

Add config.active_job.queue_adapter = :resque to /config/application.rb/

Update Procfile

If a Procfile does not already exist, create one in the app's root directory. It should be called Procfile exactly. For example Procfile.txt will not work.

In the Procfile add

resque: env TERM_CHILD=1 bundle exec environment rake resque:work  

Update Rakefile

In Rakefile, add

require 'resque/tasks'  

Add below require_relative 'config/application'

Create Resque initializer

On initialization, Resque should be configured with some settings that will be used later when jobs are enqueued.

• Create or open file /config/initializers/resque.rb
• Inform file about environment:

rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..' #Sets rails_root to the root directory of app
rails_env = ENV['RAILS_ENV'] || 'development' #Gets the currnet environment, or default to development  

• Load YAML file (built in the next section) with URL variables:

resque_config = YAML.load_file(rails_root + '/config/resque.yml')  

• Parse the URL (only important for local development):

uri = URI.parse(resque_config[rails_env])  

• Create new Redis connection, dependent on environment (local does not require password):

if rails_env === 'development'  
	Resque.redis = Redis.new(host: uri.host, port: uri.port)  
elsif rails_env === 'production'  
	resque_password = 'ebHqbhJGRsTIuwE1'  
	Resque.redis = Redis.new(url: resque_config[rails_env], password: resque_password)  
end  

Set up YAML file for initializer

Now that the initializer is set, variables need to be configured for the resque_config variable in /config/initializers/resque.rb

• Create or open /config/resque.yml
  Note: This file can be located anywhere, but note the relative URL path set in the resque.rb initializer and be sure it's specifying the correct location.
• Set up URLs for Redis, based on environment

development: localhost:6379 #this is the default Redis port on local machines  
production: redis://redis-11073.c8.us-east-1-2.ec2.cloud.redislabs.com:11073 #This is found via the Heroku interface once the Redis Cloud Addon is provisioned. Not necessary for local development.  

Set up a job

Before enqueuing or running a job, that job must be defined in the app's job folder.

• Create or open /app/jobs/job_name.rb
• Set up the job:

class JobName  
  @queue = :job_name  
  
  def self.perform(param_1, param_2, param_3)  
  	# Run any background work here.  
  	# This can include sending emails,  
  	# making API calls, or any other  
  	# time-consuming or resource-heavy  
  	# operation.  
  end  
  
end  

Enqueue a job

Before a job can be executed, it must be enqueued. To enqueue a job call:

Resque.enqueue(JobName, param_1, param_2, param_3)  

Where JobName is the name of a job class, and param_1, param_2, and param_3 are params passed to that job's self.perform method. Any number of params can be passed to the job's self.perform method, depending on what that method calls for. In this example, the method requires 3 params, so in our enqueue command 3 params are specified.

The job is now enqueued and waiting to be executed.

(Re)start local Redis server

In case the Redis server session was terminated, run redis-server in a new terminal tab to (re)start. Redis must be started because it is the engine that handles the sending and recieving of the previously created job's data. If hundreds of API calls are being made, this data is requested through Redis so that the app itself doesn't timeout, crash, or tie up the app.

Executing local jobs

Having included the line require 'resque/tasks' in the Rakefile, we can run a rake task that will continue to run and listen for jobs.
$ TERM_CHILD=1 QUEUE=* rake environment resque:work

Term child specifies the signal handling path so Resque knows how to handle long-running killed jobs.

If you have any jobs enqueued at this point, running the rake task will execute the pending jobs. If not, it will actively listen (start this in a new terminal tab) for any new jobs.

DEPLOY TO HEROKU

Specify queue and map jobs

• Open or create file /lib/tasks/resque.rake
• Specify the queue, based on the environment

require 'resque/tasks'  
task "resque:setup" => :environment do  
  ENV['QUEUE'] = '*'  
end  

• Point Heroku's job worker to Resque's job by adding the following. The resque worker stays off, but this tells Heroku to run the resque worker task (which is something like env TERM_CHILD=1 bundle exec environment rake resque:work). This starts your Heroku app's listeners to listen for resque enqueued jobs.

task "jobs:work" => "resque:work"   

Provision Redis Heroku Addon

• Install the Redis instance on Heroku

$ heroku addons:create rediscloud

• Get the Redis endpoint URL

$ heroku config:get REDISCLOUD_URL  
REDIS_URL: redis://h:[email protected]:111  

Update resque.yml

Now the production URL for Redis can be updated in the /config/resque.yml file. In this example, the new Redis URL obtained is redis://h:[email protected]:111 as returned from greping for the Heroku Redis Addon endpoint URL

• Open /config/resque.yml
• Replace:

production: redis://redis-11073.c8.us-east-1-2.ec2.cloud.redislabs.com:11073  

with:

production: redis://h:[email protected]:111  

This points the production application to the correct instance of a production Redis server to send and return job data.

At this point the app should be enqueuing and executing jobs in both local and production environments.

Note that this was set up and tested with the following (in case it ever matters)

• Ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin15]
• Rails 5.0.0.1
• gem 'resque', "~> 1.22.0"