Created
January 8, 2014 09:25
-
-
Save root-11/8314018 to your computer and use it in GitHub Desktop.
From Linkedin Python post "Python parallelism in one line - A Better Model for Day to Day Threading Tasks" Does anyone know of similar "compact" examples of handling parallelism for the case where the workers may produce tasks that are put into the FIFO queue (maybe with `while` instead of map) ?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# requires python 3.2.3 or higher | |
import concurrent.futures | |
def f(x): | |
print ("working on task: %s" %{x}) | |
if x-1 > 0 : | |
return x-1 # I want this task back in the queue. | |
def main(): | |
Queue = [x for x in range(4)] | |
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor: | |
while len(Queue)>0: | |
task = Queue.pop(0) | |
future = executor.submit(f, task) | |
result = future.result() | |
if result is not None: | |
Queue.append(result) # here the f may return a task is to be added to the queue. | |
print ("processing completed.") | |
if __name__ == "__main__": | |
main() | |
# expected output: | |
# "working on task: 1" | |
# "working on task: 2" (returns 1 to the Queue) | |
# "working on task: 3" (returns 2 to the Queue) | |
# "working on task: 4" (returns 3 to the Queue) | |
# "working on task: 1" | |
# "working on task: 2" (returns 1 to the Queue) | |
# "working on task: 3" (returns 2 to the Queue) | |
# "working on task: 1" | |
# "working on task: 2" (returns 1 to the Queue) | |
# "working on task: 1" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment