python multiprocessing share variable

在某次开发中多进程间需要频繁共享变量， 碰到了各种各样的问题，总结一下。

一、multiprocessing中的Queue共享问题

1.先看一段代码

import multiprocessing, time


q = multiprocessing.Queue()
def task(count):
    for i in xrange(count):
        q.put("%d mississippi" % i)
    return "Done"

def main():
    pool = multiprocessing.Pool()
    result = pool.apply_async(task, (1,))
    time.sleep(1)
    while not q.empty():
        print q.get()
    print result.get()

if __name__ == "__main__":
    main()

运行之后，输出结果：

0 mississippi
Done

通过以上代码我们知道进程间通信可调用multiprocessing.Queue.但会发现此时的q是global。试想如果是局部变量情况会是怎么样的呢？看下面代码：

import multiprocessing, time

def task(count):
    for i in xrange(count):
        q.put("%d mississippi" % i)
    return "Done"

def main():
    q = multiprocessing.Queue()
    pool = multiprocessing.Pool()
    result = pool.apply_async(task, (1, q)) 
    time.sleep(1)
    while not q.empty():
        print q.get()
    print result.get()

if __name__ == "__main__":
    main()

运行会发现报错。

RuntimeError: Queue objects should only be shared between processes through inheritance

解决方法1》： 在Pool初始化queue

import multiprocessing, time

def task(count):
    for i in xrange(count):
        task.q.put("%d mississippi" % i)
    return "Done"

def task_init(q):
    task.q = q 

def main():
    q = multiprocessing.Queue()
    pool = multiprocessing.Pool(None, task_init, (q,))
    result = pool.apply_async(task, (1,))
    time.sleep(1)
    while not q.empty():
        print q.get()
    print result.get()

if __name__ == "__main__":
    main()

解决方法2》：调用multiprocessing下的manager（）中的queue

import multiprocessing, time

def task(count, q): 
    for i in xrange(count):
        q.put("%d mississippi" % i)
    return "Done"

def main():
    q = multiprocessing.Manager().Queue()
    pool = multiprocessing.Pool()
    result = pool.apply_async(task, (1,q,))
    time.sleep(1)
    while not q.empty():
        print q.get()
    print result.get()

if __name__ == "__main__":
    main()

小结： 多进程间变量共享均可以调用Manager（）下的类型， A manager returned by Manager() will support types