Skip to content

Instantly share code, notes, and snippets.

@vlad-bezden
Last active March 3, 2020 15:08
Show Gist options
  • Save vlad-bezden/178af60dd82f68f6321fd0ad0e783ca5 to your computer and use it in GitHub Desktop.
Save vlad-bezden/178af60dd82f68f6321fd0ad0e783ca5 to your computer and use it in GitHub Desktop.
Run blocking code on the separate executor. Use asyncio to execute blocking io operation without blocking execution thread using "run_in_executor" function
import asyncio
from concurrent.futures.thread import ThreadPoolExecutor
from time import sleep
import logging
logging.basicConfig(
level=logging.DEBUG, format="%(asctime)s %(thread)s %(funcName)s %(message)s"
)
def long_task(t):
"""Simulate long IO bound task."""
logging.info("2. t: %s", t)
sleep(t)
logging.info("4. t: %s", t)
return t ** 2
async def main():
loop = asyncio.get_running_loop()
executor = ThreadPoolExecutor(max_workers=4)
inputs = range(1, 5)
logging.info("1.")
futures = [loop.run_in_executor(executor, long_task, i) for i in inputs]
logging.info("3.")
results = await asyncio.gather(*futures)
logging.info("5.")
for (i, result) in zip(inputs, results):
logging.info("6. Result: %s, %s", i, result)
if __name__ == "__main__":
asyncio.run(main())
"""
Output:
2020-02-20 19:20:20,884 11172 __init__ Using proactor: IocpProactor
2020-02-20 19:20:20,886 11172 main 1.
2020-02-20 19:20:20,886 15576 long_task 2. t: 1
2020-02-20 19:20:20,887 11376 long_task 2. t: 2
2020-02-20 19:20:20,887 11132 long_task 2. t: 3
2020-02-20 19:20:20,888 18440 long_task 2. t: 4
2020-02-20 19:20:20,888 11172 main 3.
2020-02-20 19:20:21,888 15576 long_task 4. t: 1
2020-02-20 19:20:22,888 11376 long_task 4. t: 2
2020-02-20 19:20:23,889 11132 long_task 4. t: 3
2020-02-20 19:20:24,889 18440 long_task 4. t: 4
2020-02-20 19:20:24,890 11172 main 5.
2020-02-20 19:20:24,890 11172 main 6. Result: 1, 1
2020-02-20 19:20:24,890 11172 main 6. Result: 2, 4
2020-02-20 19:20:24,890 11172 main 6. Result: 3, 9
2020-02-20 19:20:24,891 11172 main 6. Result: 4, 16
"""
@vlad-bezden
Copy link
Author

There are 4 blocking tasks. Each task is taking 1, 2, 3, 4 seconds to run. In a blocking environment it would of taking 10 seconds to finish. Running them in executor forces them to run in parallel on own thread. So total time took max time of all threads (4 seconds).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment