Dead lock when mix with multi threading and multi process

dead_lock.md

What are we talking about

Some times we wanted to start sub process in python. There are two packages can both do the job: multiprocessing and subprocess. But these two packages serve different purpose.

multiprocessing

subprocess

dead_lock.py

from multiprocessing import Process
from threading import Thread
from threading import Lock
import time
mutex = Lock()


def start_process(sleep=0):
    print('starting process')
    time.sleep(sleep)
    p = Process(target=child_process)
    p.start()
    return p


def child_process():
    with mutex:
        print('child process started')


def test_start_process_in_main_thread_with_intentionally_lock():
    print('This sub process will block forever')
    mutex.acquire()
    start_process()


def function_need_mutex_resource(sleep=0):
    with mutex:
        print('Yea! I got lock')
        time.sleep(sleep)
    print('Yea! I released lock')


def test_start_process_in_main_thread_with_real_world_lock():
    print('In real world, functions who need mutex '
          'will release it after they finished their job')
    function_need_mutex_resource()
    start_process()


def test_start_process_in_child():
    print('But If any child thread acquired lock,'
          'and sub process start before child thread release it.'
          'This will cause sub process block forever')
    Thread(target=function_need_mutex_resource, kwargs={'sleep': 2}).start()
    Thread(target=start_process, kwargs={'sleep': 0.5}).start()


def test_start_child_process_before_child_thread():
    print(
        'Unless you always start subprocess in main thread before you creating any child thread'
    )
    start_process()
    Thread(target=function_need_mutex_resource, kwargs={'sleep': 2}).start()

The rule should be

Do not allow any lock be acquired before you start your sub process.

The different between multiprocessing.Process and subprocess.Popen is same as fork() and exec()