Last active
March 3, 2020 15:08
-
-
Save vlad-bezden/178af60dd82f68f6321fd0ad0e783ca5 to your computer and use it in GitHub Desktop.
Run blocking code on the separate executor. Use asyncio to execute blocking io operation without blocking execution thread using "run_in_executor" function
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import asyncio | |
from concurrent.futures.thread import ThreadPoolExecutor | |
from time import sleep | |
import logging | |
logging.basicConfig( | |
level=logging.DEBUG, format="%(asctime)s %(thread)s %(funcName)s %(message)s" | |
) | |
def long_task(t): | |
"""Simulate long IO bound task.""" | |
logging.info("2. t: %s", t) | |
sleep(t) | |
logging.info("4. t: %s", t) | |
return t ** 2 | |
async def main(): | |
loop = asyncio.get_running_loop() | |
executor = ThreadPoolExecutor(max_workers=4) | |
inputs = range(1, 5) | |
logging.info("1.") | |
futures = [loop.run_in_executor(executor, long_task, i) for i in inputs] | |
logging.info("3.") | |
results = await asyncio.gather(*futures) | |
logging.info("5.") | |
for (i, result) in zip(inputs, results): | |
logging.info("6. Result: %s, %s", i, result) | |
if __name__ == "__main__": | |
asyncio.run(main()) | |
""" | |
Output: | |
2020-02-20 19:20:20,884 11172 __init__ Using proactor: IocpProactor | |
2020-02-20 19:20:20,886 11172 main 1. | |
2020-02-20 19:20:20,886 15576 long_task 2. t: 1 | |
2020-02-20 19:20:20,887 11376 long_task 2. t: 2 | |
2020-02-20 19:20:20,887 11132 long_task 2. t: 3 | |
2020-02-20 19:20:20,888 18440 long_task 2. t: 4 | |
2020-02-20 19:20:20,888 11172 main 3. | |
2020-02-20 19:20:21,888 15576 long_task 4. t: 1 | |
2020-02-20 19:20:22,888 11376 long_task 4. t: 2 | |
2020-02-20 19:20:23,889 11132 long_task 4. t: 3 | |
2020-02-20 19:20:24,889 18440 long_task 4. t: 4 | |
2020-02-20 19:20:24,890 11172 main 5. | |
2020-02-20 19:20:24,890 11172 main 6. Result: 1, 1 | |
2020-02-20 19:20:24,890 11172 main 6. Result: 2, 4 | |
2020-02-20 19:20:24,890 11172 main 6. Result: 3, 9 | |
2020-02-20 19:20:24,891 11172 main 6. Result: 4, 16 | |
""" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There are 4 blocking tasks. Each task is taking 1, 2, 3, 4 seconds to run. In a blocking environment it would of taking 10 seconds to finish. Running them in executor forces them to run in parallel on own thread. So total time took max time of all threads (4 seconds).