Celery example with tasks of varying length + -Ofair

run-worker.sh

# tested with celery[redis]==3.1.17

# to run with default configuration -- tasks will take 14 seconds to complete the 20 tasks in start_all() below

celery worker -A cluster_project.celery_app -Q tester -lINFO --concurrency=4

# to run with -Ofair -- tasks will take 10 seconds to complete

celery worker -A cluster_project.celery_app -Q tester -lINFO --concurrency=4 -Ofair

tasks.py

import time
from cluster_project import celery_app

# To run this contrived example:
# 1. start worker with -Ofair or not
# 2. call start_all() and watch celery worker output
def start_all():
  slow_task.delay()
  for i in range(19):
    fast_task.delay(i)

@celery_app.task(queue='tester')
def slow_task():
  print 'starting slow task'
  time.sleep(10)
  print 'done with slow task'

@celery_app.task(queue='tester')
def fast_task(i):
  print 'starting fast task %d' % i
  time.sleep(1)
  print 'done with fast task %d' % i

What is @celery_app here. Can you share it also. it would be great for me.

Might be nice to change the imports as follows so it's easier for anyone to run:

from celery import Celery

celery_app = Celery('tasks', broker='redis://localhost')

and run as celery worker -A tasks ...

Starting version 4.0 this behavior is by defult - http://docs.celeryproject.org/en/4.0/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy.

@w00lf thanks, was curious why it wasn't in v4 docs

Starting version 4.0 this behavior is by default - http://docs.celeryproject.org/en/4.0/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy.

Sweet!