Skip to content

Instantly share code, notes, and snippets.

@aconz2
Created May 28, 2018 04:59
Show Gist options
  • Save aconz2/84df2577fbe040051c67b750f5c2cfe9 to your computer and use it in GitHub Desktop.
Save aconz2/84df2577fbe040051c67b750f5c2cfe9 to your computer and use it in GitHub Desktop.
FIFO leak when using many processes under pypy
import multiprocessing
from subprocess import run, PIPE
import os
import gc
def worker(x):
return x * x
def n_files_open():
return len(files_open())
def is_fifo(line):
return 'FIFO' in line
def files_open():
return run(['lsof', '-p', str(os.getpid())], stdout=PIPE, universal_newlines=True, check=True).stdout.split('\n')
def n_fifos_open():
return len(list((filter(is_fifo, files_open()))))
print(n_files_open(), n_fifos_open())
with multiprocessing.Pool(maxtasksperchild=1) as p:
for result in p.imap_unordered(worker, range(10 ** 2)):
pass
print(n_files_open(), n_fifos_open())
gc.collect()
print(n_files_open(), n_fifos_open())
"""
→ python3 --version
Python 3.6.5
→ python3 fifoleak.py
26 1
62 17
62 17
→ pypy3 --version
Python 3.5.3 (fdd60ed87e941677e8ea11acf9f1819466521bf2, Apr 27 2018, 15:39:57)
[PyPy 6.0.0 with GCC 7.2.0]
→ pypy3 fifoleak.py
28 1
150 113
54 17
"""
@aconz2
Copy link
Author

aconz2 commented May 28, 2018

maxtasksperchild=1 spawns a new process each time, imap_unordered is only used here b/c by default it has a chunksize of 1, so we truly are spawning one child per iterated element. the parent process doesn't trigger a gc in a simple loop (not uncommon for coordinating process) so the FIFO used in the SimpleQueue in the Pool don't get closed promptly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment