Last active
December 19, 2015 22:59
-
-
Save gshutler/6031728 to your computer and use it in GitHub Desktop.
Prioritised queue consumption with multiple queues
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
queues = [] | |
queues << queue_consumer("high") | |
queues << queue_consumer("normal") | |
def next_message | |
# look for a message from each queue in priority order | |
queues.each do |queue| | |
found, message = queue.non_blocking_dequeue | |
return [found, message] if found | |
end | |
# block on the highest priority queue for a while to avoid pounding | |
# what is a sensible time depends on your case | |
queues.first.blocking_dequeue(500ms) | |
end | |
while true do # make this a changeable flag to stop consuming gracefully | |
found, message = next_message | |
if found | |
process_message(message) | |
ack_message(message) | |
end | |
end |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Another thought, if contention isn't your reason for prioritising (so you can have multiple consumers), you need to be careful of the prefetch configuration. Otherwise one consumer can prefetch all the "high" tasks whilst all the others work on the remaining "low" tasks.