python queue join,在队列为空之前调用join时，Python 3多处理队列死锁

I have a question understanding the queue in the multiprocessing module in python 3

This is what they say in the programming guidelines:

Bear in mind that a process that has put items in a queue will wait before

terminating until all the buffered items are fed by the “feeder” thread to

the underlying pipe. (The child process can call the

Queue.cancel_join_thread

method of the queue to avoid this behaviour.)

This means that whenever you use a queue you need to make sure that all

items which have been put on the queue will eventually be removed before the

process is joined. Otherwise you cannot be sure that processes which have

put items on the queue will terminate. Remember also that non-daemonic

processes will be joined automatically.

An example which will deadlock is the following:

from multiprocessing import Process, Queue

def f(q):

q.put('X' * 1000000)

if __name__ == '__main__':

queue = Queue()

p = Process(target=f, args=(queue,))

p.start()

p.join() # this deadlocks

obj = queue.get()

A fix here would be to swap the last two lines (or simply remove the

p.join() line).

So apparently, queue.get() should not be called after a join().

However there are examples of using queues where get is called after a join like:

import multiprocessing as mp

import random

import string

# define a example function

def rand_string(length, output):

""" Generates a random string of numbers, lower- and uppercase chars. """

rand_str = ''.join(random.choice(

string.ascii_lowercase

+ string.ascii_uppercase

+ string.digits)

for i in range(length))

output.put(rand_str)

if __name__ == "__main__":

# Define an output queue

output = mp.Queue()

# Setup a list of processes that we want to run

processes = [mp.Process(target=rand_string, args=(5, output))

for x in range(2)]

# Run processes

for p in processes:

p.start()

# Exit the completed processes

for p in processes:

p.join()

# Get process results from the output queue

results = [output.get() for p in processes]

print(results)

I've run this program and it works (also posted as a solution to the StackOverFlow question Python 3 - Multiprocessing - Queue.get() does not respond).

Could someone help me understand what the rule for the deadlock is here?

解决方案

The queue implementation in multiprocessing that allows data to be transferred between processes relies on standard OS pipes.

OS pipes are not infinitely long, so the process which queues data could be blocked in the OS during the put() operation until some other process uses get() to retrieve data from the queue.

For small amounts of data, such as the one in your example, the main process can join() all the spawned subprocesses and then pick up the data. This often works well, but does not scale, and it is not clear when it will break.

But it will certainly break with large amounts of data. The subprocess will be blocked in put() waiting for the main process to remove some data from the queue with get(), but the main process is blocked in join() waiting for the subprocess to finish. This results in a deadlock.

Here is an example where a user had this exact issue. I posted some code in an answer there that helped him solve his problem.