python线程池和进程池_python 中的进程池与线程池 -- Future 与 Executor

上面我们提到了 Executor，我们不应该自己创建 Future 对象，而是应该通过 Executor 来生成

concurrent 包中有两个类继承自 Executor，分别是：ThreadPoolExecutor -- 线程池

ProcessPoolExecutor -- 进程池

他们分别维护了一个任务队列来控制并发编程，同时隐藏大量细节，让使用者在使用中足够简单

提交任务 -- submitsubmit(fn, *args, **kwargs)

提交一个任务，返回一个 Future 对象用来接收执行结果

示例

下面的例子展示了将 15 次任务的执行提交给拥有 10 个进程的进程池，并获取返回

import logging

import os

from concurrent.futures import *

from time import sleep

def current_sleep(i):

sleep(3)

return '%s.%s over sleep' % (i, os.getppid())

if __name__ == '__main__':

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s: %(message)s')

results = []

with ThreadPoolExecutor(10) as executor:

futrues = []

for i in range(15):

futrues.append(executor.submit(current_sleep, i+1))

for futrue in futrues:

logging.info(futrue.result())

打印出了：2019-07-02 15:02:49,758 - INFO: 1.16992 over sleep

2019-07-02 15:02:49,760 - INFO: 2.964 over sleep

2019-07-02 15:02:49,760 - INFO: 3.1688 over sleep

2019-07-02 15:02:49,761 - INFO: 4.20708 over sleep

2019-07-02 15:02:49,762 - INFO: 5.25720 over sleep

2019-07-02 15:02:49,762 - INFO: 6.3792 over sleep

2019-07-02 15:02:49,765 - INFO: 7.21468 over sleep

2019-07-02 15:02:49,765 - INFO: 8.16480 over sleep

2019-07-02 15:02:49,854 - INFO: 9.13940 over sleep

2019-07-02 15:02:50,694 - INFO: 10.24184 over sleep

2019-07-02 15:02:52,758 - INFO: 11.16992 over sleep

2019-07-02 15:02:52,761 - INFO: 12.1688 over sleep

2019-07-02 15:02:52,761 - INFO: 13.964 over sleep

2019-07-02 15:02:52,762 - INFO: 14.20708 over sleep

2019-07-02 15:02:52,762 - INFO: 15.25720 over sleep

可以看到，前 10 个进程在同时返回，此后其中五个进程执行任务并在 sleep 完成执行后返回

通过迭代器提交和返回任务 -- mapmap(func, *iterables, timeout=None)

map 方法与 multiprocessing.pool.Pool 中的 map 方法是一样的，将 iterable 参数传入的可迭代对象传递给不同的进程来处理，返回所有结果收集后的可迭代对象

可以通过 timeout 参数限制任务执行的超时，一旦超时，则会触发 TimeoutError 异常

如果任务执行过程中抛出了异常，map 方法并不会将异常抛出，只有在获取结果时才会抛出

示例

下面是通过 map 方法重新编写的上述示例

import logging

import os

from concurrent.futures import *

from time import sleep

def current_sleep(i):

sleep(3)

return '%s.%s over sleep' % (i, os.getpid())

if __name__ == '__main__':

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s: %(message)s')

with ProcessPoolExecutor(10) as executor:

results = executor.map(current_sleep, range(1, 16))

for result in results:

logging.info(result)

打印出了：2019-07-02 15:11:17,526 - INFO: 1.15512 over sleep

2019-07-02 15:11:17,533 - INFO: 2.19212 over sleep

2019-07-02 15:11:17,538 - INFO: 3.6428 over sleep

2019-07-02 15:11:17,538 - INFO: 4.16516 over sleep

2019-07-02 15:11:17,542 - INFO: 5.6924 over sleep

2019-07-02 15:11:17,542 - INFO: 6.25212 over sleep

2019-07-02 15:11:17,542 - INFO: 7.22456 over sleep

2019-07-02 15:11:17,543 - INFO: 8.16536 over sleep

2019-07-02 15:11:17,623 - INFO: 9.10108 over sleep

2019-07-02 15:11:18,452 - INFO: 10.17748 over sleep

2019-07-02 15:11:20,526 - INFO: 11.15512 over sleep

2019-07-02 15:11:20,534 - INFO: 12.19212 over sleep

2019-07-02 15:11:20,539 - INFO: 13.6428 over sleep

2019-07-02 15:11:20,539 - INFO: 14.16516 over sleep

2019-07-02 15:11:20,541 - INFO: 15.22456 over sleep

虽然任务是在不同时间先后完成的，但只有当全部完成或超时才会返回

关闭进程/线程池 -- shutdownshutdown(wait=True)

关闭进程/线程池，此后进程/线程池不再接受 map 或 submit 调用，否则将触发 RuntimeError

如果 wait 为 True，则阻塞等待进程/线程池关闭后返回，否则立即返回