Created
May 28, 2018 04:59
-
-
Save aconz2/84df2577fbe040051c67b750f5c2cfe9 to your computer and use it in GitHub Desktop.
FIFO leak when using many processes under pypy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import multiprocessing | |
from subprocess import run, PIPE | |
import os | |
import gc | |
def worker(x): | |
return x * x | |
def n_files_open(): | |
return len(files_open()) | |
def is_fifo(line): | |
return 'FIFO' in line | |
def files_open(): | |
return run(['lsof', '-p', str(os.getpid())], stdout=PIPE, universal_newlines=True, check=True).stdout.split('\n') | |
def n_fifos_open(): | |
return len(list((filter(is_fifo, files_open())))) | |
print(n_files_open(), n_fifos_open()) | |
with multiprocessing.Pool(maxtasksperchild=1) as p: | |
for result in p.imap_unordered(worker, range(10 ** 2)): | |
pass | |
print(n_files_open(), n_fifos_open()) | |
gc.collect() | |
print(n_files_open(), n_fifos_open()) | |
""" | |
→ python3 --version | |
Python 3.6.5 | |
→ python3 fifoleak.py | |
26 1 | |
62 17 | |
62 17 | |
→ pypy3 --version | |
Python 3.5.3 (fdd60ed87e941677e8ea11acf9f1819466521bf2, Apr 27 2018, 15:39:57) | |
[PyPy 6.0.0 with GCC 7.2.0] | |
→ pypy3 fifoleak.py | |
28 1 | |
150 113 | |
54 17 | |
""" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
maxtasksperchild=1 spawns a new process each time, imap_unordered is only used here b/c by default it has a chunksize of 1, so we truly are spawning one child per iterated element. the parent process doesn't trigger a gc in a simple loop (not uncommon for coordinating process) so the FIFO used in the SimpleQueue in the Pool don't get closed promptly