如果您运行以下命令,这是一个有趣的注意事项:import numpy
from multiprocessing import Pool
a = numpy.arange(1000000)
pool = Pool(processes = 5)
result = pool.map(numpy.sin, a)
UnpicklingError: NEWOBJ class argument has NULL tp_new
没想到,所以发生了什么,嗯:>>> help(numpy.sin)
Help on ufunc object:
sin = class ufunc(__builtin__.object)
| Functions that operate element by element on whole arrays.
|
| To see the documentation for a specific ufunc, use np.info(). For
| example, np.info(np.sin). Because ufuncs are written in C
| (for speed) and linked into Python with NumPy's ufunc facility,
| Python's help() function finds this page whenever help() is called
| on a ufunc.
是的,numpy.sin是用c实现的,因此不能直接用它进行多处理。
所以我们必须用另一个函数来包装它
性能:import time
import numpy
from multiprocessing import Pool
def numpy_sin(value):
return numpy.sin(value)
a = numpy.arange(1000000)
pool = Pool(processes = 5)
start = time.time()
result = numpy.sin(a)
end = time.time()
print 'Singled threaded %f' % (end - start)
start = time.time()
result = pool.map(numpy_sin, a)
pool.close()
pool.join()
end = time.time()
print 'Multithreaded %f' % (end - start)
$ python perf.py
Singled threaded 0.032201
Multithreaded 10.550432
哇,也没想到,对于初学者来说,我们使用python函数有几个问题,即使它只是一个包装器,而不是一个纯c函数,还有复制值的开销,默认情况下多处理不共享数据,因此每个值都需要来回复制。
请注意,如果正确分割我们的数据:import time
import numpy
from multiprocessing import Pool
def numpy_sin(value):
return numpy.sin(value)
a = [numpy.arange(100000) for _ in xrange(10)]
pool = Pool(processes = 5)
start = time.time()
result = numpy.sin(a)
end = time.time()
print 'Singled threaded %f' % (end - start)
start = time.time()
result = pool.map(numpy_sin, a)
pool.close()
pool.join()
end = time.time()
print 'Multithreaded %f' % (end - start)
$ python perf.py
Singled threaded 0.150192
Multithreaded 0.055083
因此,我们能从中得到什么,多处理是伟大的,但我们应该总是测试和比较它有时更快,有时更慢,这取决于它如何使用。。。
假设您没有使用numpy.sin,但是我建议您首先验证另一个函数,即多处理确实会加快计算速度,可能来回复制值的开销会影响您。
不管怎样,我也相信使用pool.map是多线程代码中最好、最安全的方法。。。
我希望这能有帮助。