gistfile1.txt

-When I'm configuring in a Django app, what is the purpose of the djcelery models? Right --now tables get created, but nothing flows into them. Is this the database backend
---- replacement for Redis and RabbitMQ? Or is it something else?

- Several database tables are used:

  * Monitoring

    When you use the django-admin monitor the cluster state is written
    to the TaskState and WorkerState tables.

  * Periodic tasks

    When the database-backed schedule is used this takes the periodic task
    schedule from the PeriodicTask model, there are also several other
    helper tables (IntervalSchedule, CrontabSchedule, PeriodicTasks).

  * Task results

    The database result backend is enabled by default when using django-celery
    (this is for historical reasons, and thus for backward compatibility).

    The results are stored in the TaskMeta and TaskSetMeta models.
    *these tables are not created if another result backend is configured*.

Django apps usually do create all tables, even if they are not going to be used,
in fact it is uncommon to do like django-celery and only create the result tables
if configured to use the database result backend.

-Why do workers delete items from the queue if they are unable to process them (i.e. task -wasn't found)?

The worker discards unknown tasks, messages with encoding errors and messages that doesn't
contain the proper fields (as per the task message protocol).

If it did not ack (delete) them they would be redelivered again and again causing a loop.

There has been talk about moving these messages to a dead-letter queue, but that has
not yet been implemented.

-What is the best way to identify if a task, based on id, exists in the queue and what its -status is?

Use task states (results): http://docs.celeryproject.org/en/latest/userguide/tasks.html#task-states

- How do I know if a task id is even valid?

If what you mean to ask is, "how do I know that a task with a particular id exists?",
then the answer is that you can't.

Every unknown task id in Celery is assumed to be PENDING, this is an optimization
so that it doesn't need to write a state every time it sends a task message.
But the main point is that asking this question is almost always an anti-pattern:
whenever you think that you need to know if a task exists or not, you are solving
a problem the wrong way. See http://docs.celeryproject.org/en/latest/userguide/tasks.html#state

-What is the preferred way to run 2 celeryd processes that are intended to handle
-different tasks? (i.e. proc 1 doesn't handle tasks that proc 2 is responsible for and vice versa)

Tasks are sent to queues, so you can sort the tasks into topics, let's say "hipri" and "lopri".

To tell a worker to consume from a particular queue you can use the "-Q" argument:

    celeryd -Q hipri
    celeryd -Q hipri,lopri

To properly route the tasks you should use the CELERY_ROUTES setting:

    CELERY_ROUTES = {"tasks.important": {"queue": "hipri"},
                     "tasks.insignificant": {"queue": "lowpri"}}

See the routing user guide for more information:
http://docs.celeryproject.org/en/latest/userguide/routing.html

TIP: celeryd-multi can be used to start multiple celeryd instances:

    $ celeryd-multi start w1 w2  -Q:w1 hipri -Q:w2 hipri,lopri
    $ manage.py celeryd_multi start w1 w2 -Q:w1 hipri -Q:w2 hipri,lopri

or when using the generic init script, the following can be used in your `/etc/default/celeryd`:

    CELERYD_NODES="w1 w2"
    CELERYD_OPTS="-Q:w1 hipri -Q:w2 hipri,lopri

(celeryd-multi examples: http://docs.celeryproject.org/en/latest/reference/celery.bin.celeryd_multi.html)

-These are just a few questions I've had to start. Don't get me wrong, I got it to work ---and it's really nice in that regard. I just can't find information on the above issues. --Thanks for reaching out :-)

It is hard for people involved in a project to write good documentation for it,
as with so much knowledge about the topic it is almost impossible to know what
new users need to know.

Thank you!
I will find a way to add this to the documentation.