Skip to content

Instantly share code, notes, and snippets.

@natea

natea/gist:2640459

Created May 8, 2012 23:45
Show Gist options
  • Save natea/2640459 to your computer and use it in GitHub Desktop.
Save natea/2640459 to your computer and use it in GitHub Desktop.
Download ZIP
celery tips
Raw
gistfile1.rst

Celery

Configuring your Django project to use Celery http://celeryq.org/docs/django-celery/getting-started/first-steps-with-django.html

Run the worker server in the foreground, so we can see what’s going on without consulting the logfile:

$ python manage.py celeryd -l info

For a complete listing of the command line options available, use the help command:

$ python manage.py help celeryd

Debugging Celery

You can enable better debugging info by running this command:

$ celeryd -Q build,appserver,database,celery -l debug

celeryev - event monitoring celeryctl - sending control messages to cluster

Installing RabbitMQ on MacOSX

http://ask.github.com/celery/getting-started/broker-installation.html#installing-rabbitmq-on-os-x

Configuring RabbitMQ

Before Celery will work, it needs to have a broker to talk to. The most popular and recommended broker is RabbitMQ.

Start the RabbitMQ server with:

$ rabbitmq-server
Activating RabbitMQ plugins ...
0 plugins activated:
...
broker running

Or you can run the server in the background with:

$ rabbitmq-server -detached

Stop the server with:

$ rabbitmqctl stop

In the Celery section of the Django settings.py file, you define a user to connect to the broker. By default this is:

BROKER_USER = "guest"
BROKER_PASSWORD = "guest"

So to create the user with RabbitMQ, type the following:

$ rabbitmqctl add_user guest guest
Creating user "guest" ...
...done.

Set the permissions:

$ rabbitmqctl set_permissions -p / guest ".*" ".*" ".*"
Setting permissions for user "guest" in vhost "/" ...
...done.

"/" is the name of the vhost, which you could set with this command (when I tried to run it, it said that this vhost had already been created):

$ rabbitmqctl add_vhost myvhost

Starting up the Celery debugger

First steps using Django with Celery: http://celeryproject.org/docs/django-celery/getting-started/first-steps-with-django.html

You can run Celery in the foreground with "manage.py celeryd -l info":

$ python manage.py celeryd -l info
[2010-12-27 17:35:12,926: WARNING/MainProcess] [email protected] v2.1.4 is starting.
[2010-12-27 17:35:12,931: WARNING/MainProcess] /Users/nateaune/code/djangozoom/lib/python2.6/site-packages/celery/apps/worker.py:110: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in a production environment!
  warnings.warn("Using settings.DEBUG leads to a memory leak, "
[2010-12-27 17:35:12,931: WARNING/MainProcess]
Configuration ->
    . broker -> amqp://guest@localhost:5672/
    . queues ->
        . celery -> exchange:celery (direct) binding:celery
    . concurrency -> 2
    . loader -> djcelery.loaders.DjangoLoader
    . logfile -> [stderr]@INFO
    . events -> OFF
    . beat -> OFF
    . tasks ->
    . dzworker.tasks.add
[2010-12-27 17:35:13,118: INFO/PoolWorker-1] child process calling self.run()
[2010-12-27 17:35:13,121: INFO/PoolWorker-2] child process calling self.run()
[2010-12-27 17:35:13,124: WARNING/MainProcess] [email protected] has started.

Running Celery in production

Running Celery in daemon mode: http://celeryq.org/docs/cookbook/daemonizing.html#example-django-configuration

or using Supervisor with Django: https://github.com/ask/django-celery/blob/master/contrib/supervisord/celeryd.conf

or without Django: https://github.com/ask/celery/tree/master/contrib/supervisord/

Troubleshooting on a server

Login to the server:

$ cd /usr/local/noderabbit/dztasks/tasks
$ . ../bin/activate
(dztasks)$ celeryctl status
appserver-0001: OK
proxy-0001: OK
web-0001: OK
db-0001: OK

4 nodes online.

(dztasks)$ celeryctl inspect active
<- active
-> appserver-0001: OK
    - empty -
-> proxy-0001: OK
    - empty -
-> web-0001: OK
    - empty -
-> db-0001: OK
    - empty -

(dztasks)$ celeryctl inspect scheduled
<- scheduled
-> db-0001: OK
    - empty -
-> appserver-0001: OK
    - empty -
-> proxy-0001: OK
    - empty -
-> web-0001: OK
    - empty -

Remote debugging

Add these lines to your code:

from celery.contrib import rdb
rdb.set_trace()

Then you can connect to the pdb debugger with:

$ telnet localhost 6900

Read more at http://ask.github.com/celery/tutorials/debugging.html

Troubleshooting

Here was the traceback:

Traceback (most recent call last):

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
   response = callback(request, *callback_args, **callback_kwargs)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
   return view_func(request, *args, **kwargs)

 File "/var/www/nrweb/nrweb/usercontrol/djangozoom/decorators.py", line 39, in wrapper
   response = view_func(request, *args, **kwargs)

 File "/var/www/nrweb/nrweb/usercontrol/dz2/views.py", line 31, in dashboard
   }, RequestContext(request))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/shortcuts/__init__.py", line 20, in render_to_response
   return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/loader.py", line 188, in render_to_string
   return t.render(context_instance)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 123, in render
   return self._render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 117, in _render
   return self.nodelist.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 744, in render
   bits.append(self.render_node(node, context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 73, in render_node
   result = node.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/loader_tags.py", line 127, in render
   return compiled_parent._render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 117, in _render
   return self.nodelist.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 744, in render
   bits.append(self.render_node(node, context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 73, in render_node
   result = node.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/loader_tags.py", line 127, in render
   return compiled_parent._render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 117, in _render
   return self.nodelist.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 744, in render
   bits.append(self.render_node(node, context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 73, in render_node
   result = node.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/loader_tags.py", line 64, in render
   result = block.nodelist.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 744, in render
   bits.append(self.render_node(node, context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 73, in render_node
   result = node.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/loader_tags.py", line 64, in render
   result = block.nodelist.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 744, in render
   bits.append(self.render_node(node, context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 73, in render_node
   result = node.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/loader_tags.py", line 64, in render
   result = block.nodelist.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 744, in render
   bits.append(self.render_node(node, context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 73, in render_node
   result = node.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/defaulttags.py", line 311, in render
   return self.nodelist_true.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 744, in render
   bits.append(self.render_node(node, context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 73, in render_node
   result = node.render(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/defaulttags.py", line 227, in render
   nodelist.append(node.render(context))

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/debug.py", line 90, in render
   output = self.filter_expression.resolve(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 510, in resolve
   obj = self.var.resolve(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 653, in resolve
   value = self._resolve_lookup(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/django/template/base.py", line 698, in _resolve_lookup
   current = current()

 File "/var/www/nrweb/nrweb/usercontrol/dz2/models.py", line 512, in status
   return self.get_task_meta()['status']

 File "/var/www/nrweb/nrweb/usercontrol/dz2/models.py", line 450, in get_task_meta
   taskmeta = default_backend.get_task_meta(task_id)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/celery/backends/base.py", line 187, in get_task_meta
   meta = self._get_task_meta_for(task_id)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/celery/backends/database.py", line 63, in _get_task_meta_for
   task = session.query(Task).filter(Task.task_id == task_id).first()

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 1620, in first
   ret = list(self[0:1])

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 1528, in __getitem__
   return list(res)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 1689, in __iter__
   return self._execute_and_instances(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 1694, in _execute_and_instances
   mapper=self._mapper_zero_or_none())

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 720, in execute
   clause, params or {})

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1191, in execute
   params)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
   return self.__execute_context(context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1302, in __execute_context
   context.parameters[0], context=context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
   context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
   context)

 File "/var/www/nrweb/nrweb/usercontrol/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 299, in do_execute
   cursor.execute(statement, parameters)

TemplateSyntaxError: Caught OperationalError while rendering: (OperationalError) terminating connection due to administrator command
server closed the connection unexpectedly
       This probably means the server terminated abnormally
       before or while processing the request.
 'SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback \nFROM celery_taskmeta \nWHERE celery_taskmeta.task_id = %(task_id_1)s \n LIMIT 1 OFFSET 0' {'task_id_1': '46a65be4-fd75-456f-ac91-8ffa2f4369cc'}

Resources

Buildout recipe for installing RabbitMQ http://pypi.python.org/pypi/rod.recipe.rabbitmq

Introduction to AMQP Messaging with RabbitMQ http://www.slideshare.net/somic/introduction-to-amqp-messaging-with-rabbitmq

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment