Task Queues in redis

gistfile1.txt

Rough concept is as follows.

Setup 4 lists for each queue.
for a queue 'foobar'

'queue:foobar:low'
'queue:foobar:medium'
'queue:foobar:high'
'queue:foobar:run_at'
and optionally 'queue:foobar:failed'

for a new task we set a key in redis

task:'unique_id:run_at   ie. 'task:4eeca799397f72:1324122434' => 'my payload'
and insert the task key into a queue using LPUSH

run_at will be ignored in any queue besides run_at

to grab a task we check to see if any tasks in the run_at queue are due to be done, if so then we RPOP the task from the queue and execute it.
if not, we check for tasks on high, medium and low queues and execute them.

This technique supports coalescing if the unique id is the same for multiple tasks, since there will only be one key and value for the task which was unset after the task was grabbed.

Also this technique can be cross language if each language uses the concept in the same way... 

The failed queue is used for tasks that for one reason or another have failed.