Python并发concurrent、Future类、异步

并发、异步

• futures在python2.7中没有自带，需要自己安装。pip install futures
• 这个模块默认是进程/线程不守护的，都要等待所有的子线程或者子进程结束程序才会停止。

ThreadPoolExecutor 线程池

• submit 提交一个任务
• map 提交多个任务
• shutdown 等待子线程结束，相当于join方法

submit 方法

• 提交一个任务
• def submit(self, fn, *args, **kwargs):

# coding=utf-8
from concurrent.futures import ThreadPoolExecutor
import time


def f():
    print 'thread start'
    time.sleep(4)
    print 'thread end'
    return 'xxx'


if __name__ == '__main__':
    executor = ThreadPoolExecutor()
    result = executor.submit(f, )
    print 'future', result.result()

map 方法

• 提交多个任务
• def map(self, fn, *iterables, **kwargs):

# coding=utf-8
from concurrent.futures import ThreadPoolExecutor
import time


def f(t):
    print 'thread start %s' % t
    time.sleep(2)
    print 'thread end %s' % t
    return t


if __name__ == '__main__':
    executor = ThreadPoolExecutor(4)
    result = executor.map(f, [1, 2, 3, 4])
    executor.shutdown()  # 线程等待，等待子线程结束
    print result  # 返回生成器对象generator

callback 回调函数

• 回调函数有一个参数，future，可以在回调函数里使用future的结果

# coding=utf-8
from concurrent.futures import ThreadPoolExecutor
import time


def foo():
    print 'thread start'
    time.sleep(4)
    return 'xx'


def save(f):
    print f
    print f.result()
    print 'save'


if __name__ == '__main__':
    pool = ThreadPoolExecutor()
    for i in range(4):
        result = pool.submit(foo)
        result.add_done_callback(save)

ProcessPoolExecutor

submit 方法

# coding=utf-8
from concurrent.futures import ProcessPoolExecutor
import time


def f(name):
    print name
    time.sleep(2)
    return 'xx'


if __name__ == '__main__':
    executor = ProcessPoolExecutor()
    result = executor.submit(f, 'who are you?')
    print result  # Future实例化的对象
    while not result.done():
        pass
    else:
        print result.result()

map 方法

# coding=utf-8
from concurrent.futures import ProcessPoolExecutor
import time


def foo(x, y):
    print x, y
    time.sleep(2)
    return x, y


if __name__ == '__main__':
    pool = ProcessPoolExecutor()
    results = pool.map(foo, (1, 2, 3), (4, 5, 6))
    print results  # <generator object result_iterator at 0x0000000002FF65E8>
    for item in results:
        # 内部调用 result()方法进入阻塞，等待进程返回结果
        print item

callback 方法

• 回调函数必须要有一个参数Future对象

# coding=utf-8
from concurrent.futures import ProcessPoolExecutor
import time


def foo():
    print 'foo start'
    time.sleep(2)
    return 'foo end'


def callback_func(future):
    print future
    print 'callback_func'


if __name__ == '__main__':
    executor = ProcessPoolExecutor(4)
    for _ in range(2):
        future = executor.submit(foo)
        future.add_done_callback(callback_func)

shutdown 方法

• 等待子进程结束，在执行主进程代码

关于map传参的问题

• def map(self, fn, *iterables, **kwargs):
• 它内部是调用了itertalls模块下的izip方法，和zip差不多

# fs = [self.submit(fn, *args) for args in itertools.izip(*iterables)]

def foo(x, y, z):
    pass
    
# 如果想给foo传参
executor = ProcessPoolExecutor(4)
executor.map(foo, (1, 2, 3),(4, 5, 6),(7, 8, 9))
"""
三次x, y, z 分别为
   1 4 7
   2 5 8
   3 6 9
"""