This script demonstrates the use of multiprocessing processes (`Writer`, `Drawer`, and `Assembler`) with synchronization using `Condition` and `Event` objects to execute sequential tasks in a concurrent environment.

multiprocessing.py

import logging
from multiprocessing import Process, Condition, Event
import time

logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s (%(levelname)s): [%(processName)s] (%(funcName)s): %(message)s",
    datefmt="%H:%M:%S",
)


class Writer(Process):
    def __init__(self, condition, event):
        super().__init__(name="Writer")
        self.condition = condition
        self.event = event

    def run(self):
        with self:
            logging.info("Writing a part of the project...")
            time.sleep(2)
            logging.info("Finished writing.")
            with self.condition:
                self.condition.notify_all()
            self.event.set()  # Signal that the writer has finished

    def __enter__(self):
        logging.info("Starting work.")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        logging.info("Work completed.")


class Drawer(Process):
    def __init__(self, condition, event):
        super().__init__(name="Drawer")
        self.condition = condition
        self.event = event

    def run(self):
        with self:
            self.event.wait()  # Wait for the writer to finish
            with self.condition:
                logging.info("Writer is done! Now starting to draw...")
                time.sleep(2)
                logging.info("Finished drawing.")
                self.condition.notify_all()

    def __enter__(self):
        logging.info("Starting work.")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        logging.info("Work completed.")


class Assembler(Process):
    def __init__(self, condition, event):
        super().__init__(name="Assembler")
        self.condition = condition
        self.event = event

    def run(self):
        with self:
            self.event.wait()  # Ensure the writer and drawer have finished
            with self.condition:
                self.condition.wait()  # Ensure drawer is done
                logging.info("Drawer is done! Assembling...")
                time.sleep(2)
                logging.info("Finished assembling.")

    def __enter__(self):
        logging.info("Starting work.")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        logging.info("Work completed.")


if __name__ == '__main__':
    condition = Condition()
    event = Event()

    writer_process = Writer(condition, event)
    drawer_process = Drawer(condition, event)
    assembler_process = Assembler(condition, event)

    writer_process.start()
    drawer_process.start()
    assembler_process.start()

    writer_process.join()
    drawer_process.join()
    assembler_process.join()

    logging.info("!")


"""
2024-05-09 15:15:45 (INFO): [Writer] (__enter__): Starting work.
2024-05-09 15:15:45 (INFO): [Writer] (run): Writing a part of the project...
2024-05-09 15:15:45 (INFO): [Drawer] (__enter__): Starting work.
2024-05-09 15:15:45 (INFO): [Assembler] (__enter__): Starting work.
2024-05-09 15:15:47 (INFO): [Writer] (run): Finished writing.
2024-05-09 15:15:47 (INFO): [Writer] (__exit__): Work completed.
2024-05-09 15:15:47 (INFO): [Drawer] (run): Writer is done! Now starting to draw...
2024-05-09 15:15:49 (INFO): [Drawer] (run): Finished drawing.
2024-05-09 15:15:49 (INFO): [Drawer] (__exit__): Work completed.
2024-05-09 15:15:55 (INFO): [Assembler] (__exit__): Work completed.
"""

Python Concurrency Concepts: Processes, Threads, and Synchronization

This guide explains key concepts in Python concurrent programming, focusing on multiprocessing and threading, as demonstrated in the script we've been discussing.

1. Processes and Multiprocessing

What is a Process?

A process is an instance of a computer program that is being executed. It has its own memory space and resources.

Multiprocessing in Python

Python's multiprocessing module allows you to create and manage processes. It's useful for CPU-bound tasks and parallel processing on multi-core systems.

Key Features:

• Separate memory space for each process
• Takes advantage of multiple CPU cores
• Avoids Global Interpreter Lock (GIL) limitations

Using the Process Class

from multiprocessing import Process

class MyProcess(Process):
    def run(self):
        # Process logic here

# Usage
process = MyProcess()
process.start()
process.join()

2. Threads and Threading

What is a Thread?

A thread is a lightweight subprocess, the smallest unit of processing that can be scheduled by an operating system. Threads within a process share the same memory space.

Threading in Python

Python's threading module is used for creating and managing threads. It's useful for I/O-bound tasks.

Key Features:

• Shares memory space within a process
• Lightweight compared to processes
• Subject to the Global Interpreter Lock (GIL) in CPython

Example of Threading

import threading

def my_function():
    # Thread logic here

thread = threading.Thread(target=my_function)
thread.start()
thread.join()

3. Synchronization Mechanisms

Condition Objects

Condition objects are synchronization primitives that allow threads to wait for certain conditions to become true.

Key Features:

• Can be used to notify waiting threads when a condition is met
• Combines a Lock and a wait queue

Usage in the Script:

from multiprocessing import Condition

condition = Condition()

# Waiting for a condition
with condition:
    condition.wait()

# Notifying threads
with condition:
    condition.notify_all()

Event Objects

Event objects are simple synchronization objects that allow communication between processes or threads.

Key Features:

• Can be set, cleared, and waited upon
• Used for signaling between processes/threads

Usage in the Script:

from multiprocessing import Event

event = Event()

# Waiting for an event
event.wait()

# Setting an event
event.set()

4. Comparing Processes and Threads

Processes:

• Separate memory space
• More overhead, slower to start
• Better for CPU-bound tasks
• Not affected by the GIL

Threads:

• Shared memory space
• Less overhead, faster to start
• Better for I/O-bound tasks
• Affected by the GIL in CPython

5. When to Use Each

• Use multiprocessing for CPU-intensive tasks that benefit from parallel execution on multiple cores.
• Use threading for I/O-bound tasks where the program spends time waiting for external operations.

6. Best Practices

1. Use with statements with synchronization primitives to ensure proper acquisition and release.
2. Be cautious of deadlocks when using multiple locks.
3. Use join() to wait for processes or threads to complete.
4. Consider using higher-level abstractions like concurrent.futures for simpler task management.

Conclusion

Understanding these concepts is crucial for effective concurrent programming in Python. The script we analyzed demonstrates how to use processes, conditions, and events to create a synchronized workflow across multiple processes. By mastering these tools, you can write efficient, concurrent Python programs that take full advantage of modern multi-core systems.