Dead lock when mix with multi threading and multi process

dead_lock.md

What are we talking about

Some times we wanted to start sub process in python. There are two packages can both do the job: multiprocessing and subprocess. But these two packages serve different purpose.

multiprocessing

subprocess

dead_lock.py

from multiprocessing import Process
from threading import Thread
from threading import Lock
import time
mutex = Lock()


def start_process(sleep=0):
    print('starting process')
    time.sleep(sleep)
    p = Process(target=child_process)
    p.start()
    return p


def child_process():
    with mutex:
        print('child process started')


def test_start_process_in_main_thread_with_intentionally_lock():
    print('This sub process will block forever')
    mutex.acquire()
    start_process()


def function_need_mutex_resource(sleep=0):
    with mutex:
        print('Yea! I got lock')
        time.sleep(sleep)
    print('Yea! I released lock')


def test_start_process_in_main_thread_with_real_world_lock():
    print('In real world, functions who need mutex '
          'will release it after they finished their job')
    function_need_mutex_resource()
    start_process()


def test_start_process_in_child():
    print('But If any child thread acquired lock,'
          'and sub process start before child thread release it.'
          'This will cause sub process block forever')
    Thread(target=function_need_mutex_resource, kwargs={'sleep': 2}).start()
    Thread(target=start_process, kwargs={'sleep': 0.5}).start()


def test_start_child_process_before_child_thread():
    print(
        'Unless you always start subprocess in main thread before you creating any child thread'
    )
    start_process()
    Thread(target=function_need_mutex_resource, kwargs={'sleep': 2}).start()

This is a gist make sure I can remember the dead lock issue when try to mix multi threading with multi process.
The only safer way I found is always start sub-process in main thread before you starting any other child thread.
Or DO NOT use multiprocessing, use subprocess instead.

The rule should be

Do not allow any lock be acquired before you start your sub process.

The different between multiprocessing.Process and subprocess.Popen is same as fork() and exec()