python 底层原理processpoolexecutor_理解Python的PoolExecutor

Demo代码和引用知识点都参考自《理解Python并发编程一篇就够了|PoolExecutor篇》--董伟明或作者个人公众号Python之美， 《Python Cookbook》和Python并发编程之线程池/进程池。

ThreadPoolExecutor 和ProcessPoolExecutor分别对threading和multiprocessing进行了高级抽象，暴露出简单的统一接口。

通过ProcessPoolExecutor 来做示例。

主要提供两个方法map() 和submit()。

map() 方法主要用来针对简化执行相同的方法，如下例：

# -*- coding: utf-8 -*-

from concurrent.futures import ProcessPoolExecutor

def fib(n, test_arg):

if n > 30:

raise Exception('can not > 30, now %s' % n)

if n <= 2:

return 1

return fib(n-1, test_arg) + fib(n-2, test_arg)

def use_map():

nums = [random.randint(0, 33) for _ in range(0, 10)]

with ProcessPoolExecutor() as executor:

try:

results = executor.map(fib, nums, nums)

for num, result in zip(nums, results):

print('fib(%s) result is %s.' % (num, result))

except Exception as e:

print(e)

执行上例，输出如下，当num为30时抛出异常捕获后程序停止运行。

...

fib(19) result is 4181.

fib(11) result is 89.

fib(2) result is 1.

fib(5) result is 5.

fib(24) result is 46368.

fib(2) result is 1.

can not > 30, now 33

使用submit()方法。

# -*- coding: utf-8 -*-

from concurrent.futures import ProcessPoolExecutor, as_completed

import random

def fib(n, test_arg):

if n > 30:

raise Exception('can not > 30, now %s' % n)

if n <= 2:

return 1

return fib(n-1, test_arg) + fib(n-2, test_arg)

def use_submit():

nums = [random.randint(0, 33) for _ in range(0, 10)]

with ProcessPoolExecutor() as executor:

futures = {executor.submit(fib, n, n): n for n in nums}

for f in as_completed(futures):

try:

print('fib(%s) result is %s.' % (futures[f], f.result()))

except Exception