Skip to content

Instantly share code, notes, and snippets.

@rdegges
Last active December 10, 2015 17:58
Show Gist options
  • Save rdegges/4471470 to your computer and use it in GitHub Desktop.
Save rdegges/4471470 to your computer and use it in GitHub Desktop.
chaosq documentation.

chaosq is an add-on providing a dedicated Amazon Simple Queue Service (SQS) queue. Amazon SQS is a high performance, infinitely scalable, and highly reliable message broker, built on top of Amazon Web Services.

Using a message queue like Amazon SQS is the simplest and most fault tolerant way to decouple, distribute, and scale your Heroku applications effectively, regardless of the size of your application.

Amazon SQS is supported by a large variety of programming languages and web frameworks including:

Installing the Add-On

chaosq can be attached to a Heroku application via the CLI:

A list of all plans available can be found [here](http://addons.heroku.com/chaosq).
:::term $ heroku addons:add chaosq:test
-----> Adding chaosq to sharp-mountain-4005... done, v18 (free)

Once chaosq has been added a CHAOSQ_URL setting will be available in the app configuration and will contain the canonical URL used to access the newly provisioned Amazon SQS queue. This can be confirmed using the heroku config:get command.

:::term $ heroku config:get CHAOSQ_URL
sqs://user:[email protected]/queueid

After installing chaosq the application should be configured to fully integrate with the add-on.

Using with Ruby

If you're using vanilla Ruby, you can interact with Amazon SQS using Amazon's official AWS Ruby SDK.

To install the AWS Ruby SDK, see the official documentation.

Once you've got the AWS Ruby SDK installed, you can easily interact with your new Amazon SQS queue:

  • View stats about the queue.
  • Add messages to the queue.
  • Remove messages from the queue.
  • Modify queue settings.

The script below uses the AWS Ruby SDK to connect to your new Amazon SQS queue, add a message to the queue, then consume the message and remove it from the queue.

:::ruby
require 'cgi'
require 'uri'
require 'rubygems'
require 'aws-sdk'

##### GLOBALS
CHAOSQ_URL = URI(ENV['CHAOSQ_URL'])

AWS.config({
    :access_key_id => CGI::unescape(CHAOSQ_URL.user),
    :secret_access_key => CGI::unescape(CHAOSQ_URL.password),
    :sqs_endpoint => CHAOSQ_URL.host,
})

sqs = AWS::SQS.new

##### SQS INTERACTION
queue = sqs.queues.named(CHAOSQ_URL.path.split('/').last)
puts "Fetched queue."

puts "Total messages in queue: #{queue.approximate_number_of_messages}"

queue.send_message('hello, world!')
puts "Sent message to queue: 'hello, world!'"

puts "Total messages in queue: #{queue.approximate_number_of_messages}"

msg = queue.receive_message
puts "Retrieved message: '#{msg.body}'"

msg.delete
puts "Deleted message from queue."

puts "Total messages in queue: #{queue.approximate_number_of_messages}"

For more examples and information, the official AWS Ruby SDK documentation provides lots of example code.

Using with Python

Using SQS with vanilla Python is easy using the boto library.

To get started, you'll need to install boto using pip, the Python package manager:

::: term $ pip install -U boto

Once you've got boto installed, you can perform any standard SQS functions. The script below uses boto to connect to your new Amazon SQS queue, add a message to the queue, then consume the message and remove it from the queue.

::: python
from os import environ
from urllib import unquote
from urlparse import urlparse

from boto.sqs.connection import SQSConnection
from boto.sqs.message import Message

##### GLOBALS
CHAOSQ_URL = urlparse(environ['CHAOSQ_URL'])
conn = SQSConnection(unquote(CHAOSQ_URL.username), unquote(CHAOSQ_URL.password))

##### SQS INTERACTION
queue = conn.get_queue(CHAOSQ_URL.path.split('/')[-1])
print 'Fetched queue.'

print 'Total messages in queue: %d' % queue.count()

message = Message()
message.set_body('hello, world!')
queue.write(message)
print "Sent message to queue: 'hello, world!'"

print 'Total messages in queue: %d' % queue.count()

message = queue.get_messages()[0]
print "Retrieved message: '%s'" % message.get_body()

queue.delete_message(message)
print 'Deleted message from queue.'

For more examples and information, the official boto SQS documentation provides lots of example code.

Using with Django

You can use SQS with Django via the Celery library. Celery allows you to process tasks asynchronously, using Amazon SQS as a message broker.

Before you begin, you'll need to install Celery. Since each Django project's requirements are different, you'll need to follow the official django-celery getting started guide.

Once you've got django-celery configured to your liking, you'll need to configure your broker settings in order to make django-celery use your new Amazon SQS queue.

To do this, open your project's settings.py file, and add the following:

::: python
from os import environ
from urllib import unquote
from urlparse import urlparse

CHAOSQ_URL = urlparse(environ['CHAOSQ_URL'])
BROKER_URL = '%s://%s:%s@' % (CHAOSQ_URL.scheme, unquote(CHAOSQ_URL.username), unquote(CHAOSQ_URL.password))

Once you've got this setup, you can use Celery like you would normally to process tasks asynchronously!

For more information on django-celery and Amazon SQS, the django-celery SQS documentation contains everything you need to know.

NOTE: In order for django-celery to work properly with Amazon SQS, you'll need to have both the django-celery and boto libraries installed.

Using with Flask

Using with Java

Using with Node.js

Dashboard

Troubleshooting

Migrating Between Plans

chaosq's upgrade process is completely instant and transparent, which means you can freely upgrade or downgrade your chaosq plan at any time with no downtime or application connectivity issues.

Use the heroku addons:upgrade command to migrate to a new plan.

:::term $ heroku addons:upgrade chaosq:medium
-----> Upgrading chaosq:medium to sharp-mountain-4005... done, v18 ($49/mo)
Your plan has been updated to: chaosq:medium

Removing the Add-On

chaosq can be removed via the CLI.

This will destroy all associated data and cannot be undone!
:::term $ heroku addons:remove chaosq
-----> Removing chaosq from sharp-mountain-4005... done, v20 (free)

Support

All chaosq support and runtime issues should be submitted via one of the Heroku Support channels. Any non-support related issues or product feedback is welcome via email: [email protected]

@kennethreitz
Copy link

We should just recommend how to use with Celery, not Django (in my opinion). The new celery 3.0 interface is very simple.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment