python3多进程包_一个异步多进程python包aiomultiprocess

aiomultiprocess是一个异步多进程的python库，其依赖于aiohttp和asyncio两个库。aiohttp是一个基于asyncio的异步http客户端和服务器。asyncio 是用来编写 并发 代码的库，使用 async/await 语法。asyncio 被用作多个提供高性能 Python 异步框架的基础，包括网络和网站服务，数据库连接库，分布式任务队列等等。

On their own, AsyncIO and multiprocessing are useful, but limited: AsyncIO still can’t exceed the speed of GIL, and multiprocessing only works on one task at a time. But together, they can fully realize their true potential.

aiomultiprocess presents a simple interface, while running a full AsyncIO event loop on each child process, enabling levels of concurrency never before seen in a Python application. Each child process can execute multiple coroutines(协程) at once, limited only by the workload and number of cores available.

举个例子：

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29#coding:utf8

import asyncio

from aiohttp import request

from aiomultiprocess import Pool

url = "https://github.com"

async def request_one(json_req):

try:

print(json_req)

async with request(method='GET', url=url) as rsp:

return await rsp.text("utf-8"), json_req["uid"] #这里获取uid是为了区分返回的结果

except Exception:

print("error")

def request_batch(list_req):

async def run_all():

async with Pool(10) as pool:

#list传入请求列表，request_one单次请求函数

return await pool.map(request_one, list_req)

return asyncio.run(run_all()) #py3.7 asyncio.run() 函数用来运行最高层级的入口点 "main()" 函数

# loop = asyncio.get_event_loop()

# return loop.run_until_complete(run_all())

if __name__=="__main__":

out = request_batch([{"uid":"2"}, {"uid":"4"}])

print(len(out))

for o in out:

print(o)

asyncio.run is a Python 3.7 addition. In 3.5-3.6, your example is roughly equivalent to:

1

2

3

4

5import asyncio

futures = [...]

loop = asyncio.get_event_loop()

loop.run_until_complete(asyncio.wait(futures))

参考