Skip to content

Instantly share code, notes, and snippets.

@gshutler
Last active December 19, 2015 22:59
Show Gist options
  • Save gshutler/6031728 to your computer and use it in GitHub Desktop.
Save gshutler/6031728 to your computer and use it in GitHub Desktop.
Prioritised queue consumption with multiple queues
queues = []
queues << queue_consumer("high")
queues << queue_consumer("normal")
def next_message
# look for a message from each queue in priority order
queues.each do |queue|
found, message = queue.non_blocking_dequeue
return [found, message] if found
end
# block on the highest priority queue for a while to avoid pounding
# what is a sensible time depends on your case
queues.first.blocking_dequeue(500ms)
end
while true do # make this a changeable flag to stop consuming gracefully
found, message = next_message
if found
process_message(message)
ack_message(message)
end
end
@gshutler
Copy link
Author

Another thought, if contention isn't your reason for prioritising (so you can have multiple consumers), you need to be careful of the prefetch configuration. Otherwise one consumer can prefetch all the "high" tasks whilst all the others work on the remaining "low" tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment