python进程池卡住_Python，调用进程池而不阻塞事件循环

If I run the following code:

import asyncio

import time

import concurrent.futures

def cpu_bound(mul):

for i in range(mul*10**8):

i+=1

print('result = ', i)

return i

async def say_after(delay, what):

print('sleeping async...')

await asyncio.sleep(delay)

print(what)

# The run_in_pool function must not block the event loop

async def run_in_pool():

with concurrent.futures.ProcessPoolExecutor() as executor:

result = executor.map(cpu_bound, [1, 1, 1])

async def main():

task1 = asyncio.create_task(say_after(0.1, 'hello'))

task2 = asyncio.create_task(run_in_pool())

task3 = asyncio.create_task(say_after(0.1, 'world'))

print(f"started at {time.strftime('%X')}")

await task1

await task2

await task3

print(f"finished at {time.strftime('%X')}")

if __name__ == '__main__':

asyncio.run(main())

The output is:

started at 18:19:28

sleeping async...

result = 100000000

result = 100000000

result = 100000000

sleeping async...

hello

world

finished at 18:19:34

This shows that the event loop blocks until the cpu bound jobs (task2) finish and it continues afterwards with the task3.

If I run only one cpu bound job (the run_in_pool is the following one):

async def run_in_pool():

loop = asyncio.get_running_loop()

with concurrent.futures.ProcessPoolExecutor() as executor:

result = await loop.run_in_executor(executor, cpu_bound, 1)

Then it seems that the event loop doesn't block since the output is:

started at 18:16:23

sleeping async...

sleeping async...

hello

world

result = 100000000

finished at 18:16:28

How can I run many cpu bound jobs (in task2) in a process pool without blocking the event loop?

解决方案

As you discovered, you need to use asyncio's own run_in_executor to wait for submitted tasks to finish without blocking the event loop. Asyncio doesn't provide the equivalent of map, but it's not hard to emulate it:

async def run_in_pool():

with concurrent.futures.ProcessPoolExecutor() as executor:

futures = [loop.run_in_executor(executor, cpu_bound, i)

for i in (1, 1, 1)]

result = await asyncio.gather(*futures)