Skip to content

Instantly share code, notes, and snippets.

@jonafato
Last active July 9, 2018 01:15
Show Gist options
  • Save jonafato/95545488e8a453c5a3efd3b0c1972847 to your computer and use it in GitHub Desktop.
Save jonafato/95545488e8a453c5a3efd3b0c1972847 to your computer and use it in GitHub Desktop.
ProcessPoolExecutor bug?

I think this is a bug. Under 3.4 - 3.6, this works just fine. However, in 3.7.0 and 3.8.0a0 (current master; 86bfed372b81b8111a56a3311d537566d5df7f61), I get the following on the loop.close() call:

import asyncio
from concurrent.futures import ProcessPoolExecutor
import os

loop = asyncio.get_event_loop_policy().new_event_loop()
loop.set_default_executor(ProcessPoolExecutor())
future = loop.run_in_executor(None, os.getpid)
loop.run_until_complete(future)
loop.close()

Exception ignored in: <function ProcessPoolExecutor._start_queue_management_thread.<locals>.weakref_cb at 0x7fa1201bc268>
Traceback (most recent call last):
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/concurrent/futures/process.py", line 567, in weakref_cb
    thread_wakeup.wakeup()
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/concurrent/futures/process.py", line 89, in wakeup
    self._writer.send_bytes(b"")
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/multiprocessing/connection.py", line 183, in send_bytes
    self._check_closed()
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/multiprocessing/connection.py", line 136, in _check_closed
Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/concurrent/futures/process.py", line 368, in _queue_management_worker
    thread_wakeup.clear()
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/concurrent/futures/process.py", line 92, in clear
    while self._reader.poll():
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/multiprocessing/connection.py", line 255, in poll
    self._check_closed()
  File "/nix/store/y011099siz87niibp863wywmrqmiabb3-python3-3.7.0/lib/python3.7/multiprocessing/connection.py", line 136, in _check_closed
    raise OSError("handle is closed")
OSError: handle is closed

    raise OSError("handle is closed")
OSError: handle is closed

After this error, I need to ^C after calling quit() to get the process to exit.

@jonafato
Copy link
Author

jonafato commented Jul 7, 2018

Credit where due, this comes from a test in pytest-asyncio.

@aaronchall
Copy link

Works ok pasting it all at once, and line-by-line - for me:

$ python
Python 3.6.4 |Anaconda custom (64-bit)| (default, Mar 13 2018, 01:15:57) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> import asyncio
>>> from concurrent.futures import ProcessPoolExecutor
>>> import os
>>> 
>>> loop = asyncio.get_event_loop_policy().new_event_loop()
>>> loop.close()
>>> loop = asyncio.get_event_loop_policy().new_event_loop()
>>> loop.set_default_executor(ProcessPoolExecutor())
>>> loop.close()
>>> loop = asyncio.get_event_loop_policy().new_event_loop()
>>> loop.set_default_executor(ProcessPoolExecutor())
>>> future = loop.run_in_executor(None, os.getpid)
>>> loop.run_until_complete(future)
21134
>>> loop.close()
>>> quit()
$ 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment