Python 多进程编程

进程池

with Pool(processes=4) as pool:

    # print "[0, 1, 4,..., 81]", 循序返回
    print(pool.map(f, range(10)))

    # print same numbers in arbitrary order
    for i in pool.imap_unordered(f, range(10)):
        print(i)

In [14]: class Foo(): 
    ...:     @staticmethod 
    ...:     def work( x): 
    ...:         return x * x 
    ...:          
    ...:                                                                        

In [15]:                                                                        

In [15]: with Pool() as pool: 
    ...:     a = Foo() 
    ...:     print(pool.map(a.work, range(10))) 
    ...:      
    ...:                                                                        
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Process

from multiprocessing import Process, Queue

def send(queue):
    data = [1, 2, 3, 4, 5, None]
    for item in data:
        queue.put(item)
        import time
        time.sleep(1)
    print("send complated")


def receive(queue):
    result = list()
    while True:
        item = queue.get()
        if item is None:
            break

        result.append(item)
        print("receive item: ", item)
    print("Result is ", result)


if __name__ == "__main__":
    q = Queue()
    p_send = Process(target=send, args=(q, ))
    p_receive = Process(target=receive, args=(q, ))
    p_send.start()
    p_receive.start()

进程间通信

使用Queue，代码如上

常见问题

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

解决方案: https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function

以下是部分翻译.

这个问题的原因是Pool会使用queue.Queue传递任务到worker进程. 所有走queue.Queue的对象必须可以被序列化. 但是类中的函数不能被序列化, 所以会报错. 在Python3中只有顶层的函数可以被序列化.

解决方案:

• 将函数提炼到顶层, 如果在类中的函数可以用如下的方式来提炼:

class A:
    @staticmethod
    def square(x):
        return x * x
                
def worker(x):
    return A.square(x)

• 使用更加强大的序列化工具, 如dill
• 使用封装了dill的多进程调度包, 如pathos

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> p = Pool(4)
>>> class Test(object):
...   def plus(self, x, y): 
...     return x+y
... 
>>> t = Test()
>>> p.map(t.plus, x, y)
[4, 6, 8, 10]
>>> 
>>> class Foo(object):
...   @staticmethod
...   def work(self, x):
...     return x+1
... 
>>> f = Foo()
>>> p.apipe(f.work, f, 100)
<processing.pool.ApplyResult object at 0x10504f8d0>
>>> res = _
>>> res.get()
101