[python] pool

http://forlinux.blog.51cto.com/8001278/1423390

from multiprocessing import Pool

apply 开启多个进程并发执行

apply_async 同上，但是这个是异步的，非阻塞的。

map 类似于内建函数map，后面提供的参数列表会一个一个应用于函数,。这里会开发多个进程并发一起执行。

map_async 和map相同，只不过这是一个异步的，不会阻塞等待结果。该函数会返回一个结果对象。

---

https://docs.python.org/2/library/multiprocessing.html

apply(func[, args[, kwds]])

Equivalent of the apply() built-in function. It blocks until the result is ready, so apply_async() is better suited for performing work in parallel. Additionally, func is only executed in one of the workers of the pool.

apply_async(func[, args[, kwds[, callback]]])

A variant of the apply() method which returns a result object.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it (unless the call failed). callback should complete immediately since otherwise the thread which handles the results will get blocked.

map(func, iterable[, chunksize])

A parallel equivalent of the map() built-in function (it supports only one iterable argument though). It blocks until the result is ready.

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

map_async(func, iterable[, chunksize[, callback]])

A variant of the map() method which returns a result object.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it (unless the call failed). callback should complete immediately since otherwise the thread which handles the results will get blocked.

---

It blocks until the result is ready.

    import multiprocessing  
    import time  

    def f(x):  
        time.sleep(2)  
        print x  

    if __name__ == '__main__':   
      pool = multiprocessing.Pool(processes=5)   
      pool.map(f, xrange(10))  
      print("end")

暂时的理解是如果调用了map, 程序会阻塞在pool.map(f, xrange(10))这一行,直到它全部执行完才会 print("end")

---

http://heipark.iteye.com/blog/2081423
1.使用map方法

    import multiprocessing  
    import time  

    def f(x):  
        time.sleep(2)  
        print x  

    if __name__ == '__main__':   
      pool = multiprocessing.Pool(processes=5)   
      pool.map(f, xrange(10))  

并发5个进程
map方法会依次将参数二数组每个元素传入参数1方法中
如果将map()替换为map_async()，则方法不会阻塞，而是直接执行main进程后面的代码，所以要配合pool.close()和pool.join()一起使用。close()方法是使pool不再接受新任务；join()方法是阻塞main进程等待子进程执行完成才可以运行后面code

2.使用apply方法

    import multiprocessing  
    import time  

    def func(msg):  
        print msg  
        time.sleep(1)  

    if __name__ == "__main__":  
        print 'start main-process'  
        pool = multiprocessing.Pool(processes=4)  
        for i in xrange(10):  
            msg = "hello %d" %(i)  
            pool.apply_async(func, (msg, ))  
        pool.close()  
        pool.join()  
        print "Sub-process(es) done."  
        print 'Main-process done.'  

apply_async和apply方法区别是前者不会阻塞main进程，需要用pool.close()和join()方法阻塞等待子进程执行。
apply与map方法相比，它只是调用方法和参数，而map方法会将数组参数迭代传给被调用方法
pool.apply_async(func, (msg, )) 这行msg后面的逗号是不能省略的，否则不会执行func方法

---

http://stackoverflow.com/questions/8533318/python-multiprocessing-pool-when-to-use-apply-apply-async-or-map

Pool.apply blocks until the function is completed.

Pool.apply_async is also like Python’s built-in apply, except that the call returns immediately instead of waiting for the result. An ApplyResult object is returned. You call its get() method to retrieve the result of the function call. The get() method blocks until the function is completed. Thus, pool.apply(func, args, kwargs) is equivalent to pool.apply_async(func, args, kwargs).get().

In contrast to Pool.apply, the Pool.apply_async method also has a callback which, if supplied, is called when the function is complete. This can be used instead of calling get().

For example:

import multiprocessing as mp
import time

def foo_pool(x):
    time.sleep(2)
    return x*x

result_list = []
def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

def apply_async_with_callback():
    pool = mp.Pool()
    for i in range(10):
        pool.apply_async(foo_pool, args = (i, ), callback = log_result)
    pool.close()
    pool.join()
    print(result_list)

if __name__ == '__main__':
    apply_async_with_callback()

may yield a result such as

[1, 0, 4, 9, 25, 16, 49, 36, 81, 64]

Notice, unlike pool.map, the order of the results may not correspond to the order in which the pool.apply_async calls were made.

So, if you need to run a function in a separate process, but want the current process to block until that function returns, use Pool.apply. Like Pool.apply, Pool.map blocks until the complete result is returned.

If you want the Pool of worker processes to perform many function calls asynchronously, use Pool.apply_async. The order of the results is not guaranteed to be the same as the order of the calls to Pool.apply_async.

Notice also that you could call a number of different functions with Pool.apply_async (not all calls need to use the same function).

In contrast, Pool.map applies the same function to many arguments. However, unlike Pool.apply_async, the results are returned in an order corresponding to the order of the arguments.