python并行计算numpy_如何在python numpy中并行化求和计算？

我想出了如何通过多处理、应用异步和回调来并行化一个数组的总和，所以我在这里为其他人发布这个。我使用the example page for Parallel Python作为Sum回调类，尽管实际上我并没有使用该包来实现。不过，这给了我使用回调的想法。这是我最终使用的简化代码，它做了我想做的事情。import multiprocessing

import numpy as np

import thread

class Sum: #again, this class is from ParallelPython's example code (I modified for an array and added comments)

def __init__(self):

self.value = np.zeros((1,512*512)) #this is the initialization of the sum

self.lock = thread.allocate_lock()

self.count = 0

def add(self,value):

self.count += 1

self.lock.acquire() #lock so sum is correct if two processes return at same time

self.value += value #the actual summation

self.lock.release()

def computation(index):

array1 = np.ones((1,512*512))*index #this is where the array-returning computation goes

return array1

def summers(num_iters):

pool = multiprocessing.Pool(processes=8)

sumArr = Sum() #create an instance of callback class and zero the sum

for index in range(num_iters):

singlepoolresult = pool.apply_async(computation,(index,),callback=sumArr.add)

pool.close()

pool.join() #waits for all the processes to finish

return sumArr.value

我还可以使用一个并行化的映射来完成这项工作，这是在另一个答案中提出的。我之前试过，但没有正确地实现。这两种方法都有效，我认为this answer很好地解释了使用哪种方法(map或apply.async)的问题。对于映射版本，不需要定义类和，summers函数变成def summers(num_iters):

pool = multiprocessing.Pool(processes=8)

outputArr = np.zeros((num_iters,1,512*512)) #you wouldn't have to initialize these

sumArr = np.zeros((1,512*512)) #but I do to make sure I have the memory

outputArr = np.array(pool.map(computation, range(num_iters)))

sumArr = outputArr.sum(0)

pool.close() #not sure if this is still needed since map waits for all iterations

return sumArr