Requests are handled by flask
, a bunch of urls are inserted in the object store (redis
)
and arguments are put on the queue (redis
again) for workers to consume. More workers
would mean more items processed in parallel.
Other possible implementations:
multiprocessing
module for consuming all cpus.multiprocessing.managers.SyncManager
for distributing task to other machines.