现象
multiprocessing.Pool传递一个普通方法(不在class中定义的)时, 能正常工作.
from multiprocessing import Pool
p = Pool(3)
def f(x):
return x*x
p.map(f, [1,2,3])
但在class中定义的方法使用multiprocessing.Pool会报pickling error错误.
报错代码
# coding: utf8
import multiprocessing
class MyTask(object):
def task(self, x):
return x*x
def run(self):
pool = multiprocessing.Pool(processes=3)
a = [1, 2, 3]
pool.map(self.task, a)
if __name__ == '__main__':
t = MyTask()
t.run()
会出现如下异常:
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
原因:
stackoverflow上的解释:
Pool
methods all use a queue.Queue
to pass tasks to the worker processes. Everything that goes through the queue.Queue
must be pickable. So, multiprocessing
can only transfer Python objects to worker processes which can be pickled. Functions are only picklable if they are defined at the top-level of a module, bound methods are not picklable.
pool
方法都使用了queue.Queue将task传递给工作进程。multiprocessing
必须将数据序列化以在进程间传递。方法只有在模块的顶层时才能被序列化,跟类绑定的方法不能被序列化,就会出现上面的异常。
解决方法:
- 用线程替换进程
- 可以使用copy_reg来规避上面的异常.
dill
或pathos.multiprocesssing
:use pathos.multiprocesssing, instead of multiprocessing. pathos.multiprocessing is a fork of multiprocessing that uses dill. dill can serialize almost anything in python, so you are able to send a lot more around in parallel.
正确代码1
# coding: utf8
from multiprocessing.pool import ThreadPool as Pool
class MyTask(object):
def task(self, x):
return x*x
def run(self):
pool = Pool(3)
a = [1, 2, 3]
ret = pool.map(self.task, a)
print ret
if __name__ == '__main__':
t = MyTask()
t.run()
正确代码2:
# coding: utf8
import multiprocessing
import types
import copy_reg
def _pickle_method(m):
if m.im_self is None:
return getattr, (m.im_class, m.im_func.func_name)
else:
return getattr, (m.im_self, m.im_func.func_name)
copy_reg.pickle(types.MethodType, _pickle_method)
class MyTask(object):
def __init__(self):
self.__result = []
def task(self, x):
return x * x
def result_collector(self, result):
self.__result.append(result)
def run(self):
pool = multiprocessing.Pool(processes=3)
a = [1, 2, 3]
ret = pool.map(self.task, a)
print ret
if __name__ == '__main__':
t = MyTask()
t.run()