Celery future concurrency document.

future.rst

Concurrency with Future

Introduction

Baseding on greenlet's concurrency mechanism, celery could interact with tornado and execute tasks in tornado. With this new feature, you can write a task as the following.

from celery import Celery
from tornado import gen, httpclient

app = Celery()

@app.task
@gen.coroutine
def post_url(url):
    res = yield httpclient.AsyncHTTPClient().fetch(url, method='POST')
    if res.code != 200:
        raise Exception(res.reason)

    return res.body

The basic idea here is combining greenlet and tornado io loop. In greenlet environment, task is dispatched to an another daemon thread, which runs the tornado ioloop. Meanwhile, in the task pool thread, current task greenlet yields to another greenlet. When the task is finished in tornado, resuming the previous task greenlet and making the greenlet finished.

Now, it only supports gevent environment. If you want to disable this feature under gevent environment, you need to uninstall tornado package. It's no elegant, but maybe practical. If distinguishing the task, its execution process may be divided into two different threads. This will destroy the context's integrity, and cause undefined behaviour.

For more details, go to #3757.