Multiprocessing Event Object In Python

Need for an Event Object

A process is a running instance of a computer program.

Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create new child processes in our program in order to execute code concurrently.

Python provides the ability to create and manage new processes via the multiprocessing.Process class.

You can learn more about multiprocessing in the tutorial:

• Multiprocessing in Python: The Complete Guide

In concurrent programming, sometimes we need to coordinate processes with a boolean variable. This might be to trigger an action or signal some result.

This could be achieved with a mutual exclusion lock (mutex) and a boolean variable, but provides no way for processes to wait for the variable to be set True.

Instead, this can be achieved using an event object.

What is an event object and how can we use it with processes in Python?

Example of Using a Shared Event with Processes

We can explore how to use a multiprocessing.Event object.

In this example we will create a suite of processes that each will perform some processing and report a message. All processes will use an event to wait to be set before starting their work. The main process will set the event and trigger the child processes to start work.

First, we can define a target task function that takes the shared multiprocessing.Event instance and a unique integer to identify the process.

1 2 3

# target task function def task(event, number): # ...

Next, the function will wait for the event to be set before starting the processing work.

1 2 3 4

... # wait for the event to be set print(f'Process {number} waiting...', flush=True) event.wait()

Once triggered, the process will generate a random number, block for a moment and report a message.

1 2 3 4 5

... # begin processing value = random() sleep(value) print(f'Process {number} got {value}', flush=True)

Tying this together, the complete target task function is listed below.

1 2 3 4 5 6 7 8 9

# target task function def task(event, number):     # wait for the event to be set     print(f'Process {number} waiting...', flush=True)     event.wait()     # begin processing     value = random()     sleep(value)     print(f'Process {number} got {value}', flush=True)

The main process will first create the shared multiprocessing.Event instance, which will be in the “not set” state by default.

1 2 3

... # create a shared event object event = Event()

Next, we can create and configure five new processes specifying the target task() function with the event object and a unique integer as arguments.

This can be achieved in a list comprehension.

1 2 3

... # create a suite of processes processes = [Process(target=task, args=(event, i)) for i in range(5)]

We can then start all child processes.

1 2 3 4

... # start all processes for process in processes:     process.start()

Next, the main process will block for a moment, then trigger the processing in all of the child processes via the event object.

1 2 3 4 5 6

... # block for a moment print('Main process blocking...') sleep(2) # trigger all child processes event.set()

The main process will then wait for all child processes to terminate.

1 2 3 4

... # wait for all child processes to terminate for process in processes:     process.join()

Tying this all together, the complete example is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

# SuperFastPython.com # example of using an event object with processes from time import sleep from random import random from multiprocessing import Process from multiprocessing import Event # target task function def task(event, number):     # wait for the event to be set     print(f'Process {number} waiting...', flush=True)     event.wait()     # begin processing     value = random()     sleep(value)     print(f'Process {number} got {value}', flush=True) # entry point if __name__ == '__main__':     # create a shared event object     event = Event()     # create a suite of processes     processes = [Process(target=task, args=(event, i)) for i in range(5)]     # start all processes     for process in processes:         process.start()     # block for a moment     print('Main process blocking...')     sleep(2)     # trigger all child processes     event.set()     # wait for all child processes to terminate     for process in processes:         process.join()

Running the example first creates and starts five child processes.

Each child process waits on the event before it starts its work, reporting a message that it is waiting.

The main process blocks for a moment, allowing all child processes to begin and start waiting on the event.

The main process then sets the event. This triggers all five child processes that perform their simulated work and report a message.

Note, your specific results will differ given the use of random numbers.

1 2 3 4 5 6 7 8 9 10 11

Main process blocking... Process 0 waiting... Process 1 waiting... Process 2 waiting... Process 3 waiting... Process 4 waiting... Process 0 got 0.06198821143561384 Process 4 got 0.219334069761699 Process 3 got 0.7335552378594119 Process 1 got 0.7948771419640999 Process 2 got 0.8713839353896263

Further Reading

This section provides additional resources that you may find helpful.

Python Multiprocessing Books

• Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Multiprocessing API Cheat Sheet

I would also recommend specific chapters in the books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism
• PEP 371 - Addition of the multiprocessing package

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps

Takeaways

You now know how to use a multiprocessing.Event Object in Python

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer

---