Python网络请求、JSON转换、多线程、异步IO

最新推荐文章于 2024-08-15 18:54:23 发布

Python_xiaowu

最新推荐文章于 2024-08-15 18:54:23 发布

阅读量307

点赞数 1

文章标签： python 开发语言后端

本文链接：https://blog.csdn.net/Python_xiaowu/article/details/121546267

版权

在 Python 众多的 HTTP 客户端中，最有名的莫过于、和。requestsaiohttphttpx

在不借助其他第三方库的情况下，只能发送同步请求；只能发送异步请求；既能发送同步请求，又能发送异步请求。requestsaiohttphttpx

那么怎么选择呢

只发同步请求用，但可配合多线程变异步。requests
只发异步请求用，但可以配合await变同步。aiohttp
httpx可以发同步请求也可以异步，但是请求速度同步略差于，异步略差于requestsaiohttp

Asyncio 的强大。但是，任何一种方案都不是完美的，都存在一定的局限性，Asyncio 同样如此。

实际使用中，想用好 Asyncio，特别是发挥其强大的功能，很多情况下必须得有相应的 Python 库支持。

比如

requests 库并不兼容 Asyncio，而 aiohttp 库兼容。

请求

这里先说requests

安装依赖

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>pip</strong></span> install requests
</span></span></span></span>

响应

响应的类型

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#1f7199">#获取接口返回的字符串数据</span>
r.<span style="color:#333333"><strong>text</strong></span>

<span style="color:#1f7199">#获取接口返回的json数据，即直接将json格式的数据转换为json对象</span>
r.json()

<span style="color:#1f7199">#获取接口返回的二进制数据，假设二进制数据如果为图片可以继续转换成图片</span>
r.content

<span style="color:#1f7199">#获取原始套接字,使用r.raw请在 requests 请求中加上参数 stream=True</span>
r.raw
</span></span></span></span>

获取请求响应的其他信息

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6">#获取状态码
<span style="color:#333333"><strong>r</strong></span><span style="color:#880000">.status_code</span>

#获取请求的<span style="color:#333333"><strong>url</strong></span>
<span style="color:#333333"><strong>r</strong></span><span style="color:#880000">.url</span>

#获取指定<span style="color:#333333"><strong>cookies</strong></span>信息
<span style="color:#333333"><strong>r</strong></span><span style="color:#880000">.cookies</span><span style="color:#bc6060">['token']</span>

#获取访问服务器返回给我们的响应头部信息
<span style="color:#333333"><strong>r</strong></span><span style="color:#880000">.headers</span>

#获取指定访问服务器返回给我们的响应头部信息
<span style="color:#333333"><strong>r</strong></span><span style="color:#880000">.headers</span><span style="color:#bc6060">['Content-Type']</span>

#获取发送到服务器的请求的头部的信息
<span style="color:#333333"><strong>r</strong></span><span style="color:#880000">.request</span><span style="color:#880000">.headers</span>
</span></span></span></span>

请求

获取请求

获取请求：

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>res</strong></span> = requests.<span style="color:#397300">get</span>(url,data=data,cookies=cookie,headers=header,verify=False,<span style="color:#333333"><strong>files</strong></span>=<span style="color:#333333"><strong>file</strong></span>)
</span></span></span></span>

data可传可不传，data是字典格式。

如果url是https的话，加上verify=False。如果url是http的话，可不加。

示例1

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> requests

<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    r = requests.<span style="color:#333333"><strong>get</strong></span>(<span style="color:#880000">"https://www.psvmc.cn"</span>)
    <span style="color:#397300">print</span>(r.text)
</span></span></span></span>

示例2

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> requests

<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    data = {<span style="color:#880000">'username'</span>: <span style="color:#880000">'admin'</span>, <span style="color:#880000">'passwd'</span>: <span style="color:#880000">'123456'</span>}
    r = requests.<span style="color:#333333"><strong>get</strong></span>(<span style="color:#880000">"https://www.psvmc.cn/login.json"</span>, params=data)
    <span style="color:#397300">print</span>(r.status_code)
    <span style="color:#397300">print</span>(r.json()[<span style="color:#880000">"obj"</span>])
</span></span></span></span>

POST请求

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6">url_post = "https://www.psvmc.cn/login.json"

<span style="color:#888888">#不包含任何参数的请求</span>
r = requests.post(url_post)

<span style="color:#888888">#不包含任何参数的请求,设置超时10s，timeout不设置则默认60s</span>
r = requests.post(url_post,timeout=10)

<span style="color:#888888">#携带参数的请求，dict_param为参数字典，默认data=dict_param，使用data=则表示post的是form请求</span>
<span style="color:#888888">#即 application/x-www-form-urlencoded 。</span>
r = requests.post(url_post, data=dict_param)

<span style="color:#888888">#携带参数的请求，dict_param为参数字典，使用json=则表示post的是json请求</span>
r = requests.post(url_post, json=dict_param)

<span style="color:#888888">#携带参数的请求，body传字符串，这里是JSON字符串。</span>
r = requests.post(url_post, data=json.dumps(dict_param))

<span style="color:#888888">#携带参数的请求，dict_param为参数字典，设置超时10s,并携带headers属性</span>
r = requests.post(
    url_post,
    data=dict_param,
    timeout=10,
    headers={
        'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X)'
    })

<span style="color:#888888">#post请求上传文件</span>
url = 'http://apihost/upload/post'
files = {'file': open('report.xls', 'rb')}
r = requests.post(url, files=files)
</span></span></span></span>

其他类型请求

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6">r = requests.put(url, data =dict_param)
r = requests.delete(url)
r = requests.head(url)
r = requests.options(url)
</span></span></span></span>

代理

跨域的时候可以考虑代理访问，不管是post请求还是get请求，只需要添加代理即可。

客户端开发时不用考虑跨域问题，没有必要设置代理访问。

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6">proxies = {
  <span style="color:#880000">"http"</span>: <span style="color:#880000">"http://10.10.1.10:3128"</span>,
  <span style="color:#880000">"https"</span>: <span style="color:#880000">"http://10.10.1.10:1080"</span>,
}
requests.get(url_get, proxies=proxies)
</span></span></span></span>

查看代理是否有效

和telnet作用一样

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> telnetlib

<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    <span style="color:#333333"><strong>try</strong></span>:
        telnetlib.Telnet(<span style="color:#880000">'110.242.68.4'</span>, port=<span style="color:#880000">'80'</span>, timeout=<span style="color:#880000">3</span>)
    <span style="color:#333333"><strong>except</strong></span>:
        print(<span style="color:#880000">'ip无效！'</span>)
    <span style="color:#333333"><strong>else</strong></span>:
        print(<span style="color:#880000">'ip有效！'</span>)
</span></span></span></span>

异步请求

aiohttp 的代码与 httpx 异步模式的代码重合度90%，只不过把换成了，AsyncClientClientSession

另外，在使用 httpx 时，当你时就已经发送了请求。但是当使用时，只有在时才会真正发送请求。await client.postaiohttpawiat resp.json()

aiohttp

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> aiohttp
<span style="color:#333333"><strong>import</strong></span> asyncio


<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    <span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>with</strong></span> aiohttp.ClientSession() <span style="color:#333333"><strong>as</strong></span> client:
        resp = <span style="color:#333333"><strong>await</strong></span> client.post(<span style="color:#880000">'https://www.psvmc.cn/login.json'</span>,
                                 json={<span style="color:#880000">'ts'</span>: <span style="color:#880000">'2020-01-20 13:14:15'</span>})
        result = <span style="color:#333333"><strong>await</strong></span> resp.json()
        print(result)


asyncio.run(main())
</span></span></span></span>

断续器

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> httpx
<span style="color:#333333"><strong>import</strong></span> asyncio


<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    <span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>with</strong></span> httpx.AsyncClient() <span style="color:#333333"><strong>as</strong></span> client:
        resp = <span style="color:#333333"><strong>await</strong></span> client.post(<span style="color:#880000">'https://www.psvmc.cn/login.json'</span>,
                                 json={<span style="color:#880000">'ts'</span>: <span style="color:#880000">'2020-01-20 13:14:15'</span>})
        result = resp.json()
        print(result)


asyncio.run(main())
</span></span></span></span>

断续器

字符串转对象

import json

# 一些 JSON:
x =  '{ "name":"Bill", "age":63, "city":"Seatle"}'

# 解析 x:
y = json.loads(x)

# 结果是 Python 字典：
print(y["age"])

对象转字符串

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> json

<span style="color:#888888"># Python 对象（字典）：</span>
x = {
  <span style="color:#880000">"name"</span>: <span style="color:#880000">"Bill"</span>,
  <span style="color:#880000">"age"</span>: <span style="color:#880000">63</span>,
  <span style="color:#880000">"city"</span>: <span style="color:#880000">"Seatle"</span>
}

<span style="color:#888888"># 转换为 JSON：</span>
y = json.dumps(x)

<span style="color:#888888"># 结果是 JSON 字符串：</span>
print(y)
</span></span></span></span>

当 Python 转换为 JSON 时，Python 对象会被转换为 JSON（JavaScript）等效项：

蟒	断续器
字典	对象
列表	数组
元	数组
str	字符串
int	数
浮	数
真	真
假	假
没有	零

异步IO（协程）和请求

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>pip</strong></span> install aiohttp
</span></span></span></span>

简单示例

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> asyncio


<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>test</strong></span>():
    <span style="color:#333333"><strong>await</strong></span> asyncio.sleep(<span style="color:#880000">3</span>)
    <span style="color:#333333"><strong>return</strong></span> <span style="color:#880000">"123"</span>


<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    result = <span style="color:#333333"><strong>await</strong></span> test()
    print(result)


<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    asyncio.run(main())
</span></span></span></span>

异步请求

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> asyncio
<span style="color:#333333"><strong>import</strong></span> aiohttp
<span style="color:#333333"><strong>import</strong></span> time


<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>download_one</strong></span>(url):
    <span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>with</strong></span> aiohttp.ClientSession() <span style="color:#333333"><strong>as</strong></span> session:
        <span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>with</strong></span> session.get(url) <span style="color:#333333"><strong>as</strong></span> resp:
            print(<span style="color:#880000">'Read {} from {}'</span>.format(resp.content_length, url))


<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>download_all</strong></span>(sites):
    tasks = [asyncio.ensure_future(download_one(site)) <span style="color:#333333"><strong>for</strong></span> site <span style="color:#333333"><strong>in</strong></span> sites]
    <span style="color:#333333"><strong>await</strong></span> asyncio.gather(*tasks)


<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    sites = [
        <span style="color:#880000">'https://www.psvmc.cn/index.html'</span>,
        <span style="color:#880000">'https://www.psvmc.cn/login.json'</span>,
        <span style="color:#880000">'https://www.psvmc.cn/userlist.json'</span>
    ]
    start_time = time.perf_counter()

    loop = asyncio.get_event_loop()
    <span style="color:#333333"><strong>try</strong></span>:
        loop.run_until_complete(download_all(sites))
    <span style="color:#333333"><strong>finally</strong></span>:
        <span style="color:#333333"><strong>if</strong></span> loop.is_running():
            loop.close()
    end_time = time.perf_counter()
    print(<span style="color:#880000">'Download {} sites in {} seconds'</span>.format(
        len(sites), end_time - start_time))


<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    main()
</span></span></span></span>

多线程

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>from</strong></span> concurrent.futures <span style="color:#333333"><strong>import</strong></span> ThreadPoolExecutor
<span style="color:#333333"><strong>import</strong></span> threading


<span style="color:#888888"># 定义一个准备作为线程任务的函数</span>
<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>action</strong></span>(num):
    print(threading.current_thread().name)
    <span style="color:#333333"><strong>return</strong></span> num+<span style="color:#880000">100</span>


<span style="color:#888888"># 创建一个包含4条线程的线程池</span>
<span style="color:#333333"><strong>with</strong></span> ThreadPoolExecutor(max_workers=<span style="color:#880000">3</span>) <span style="color:#333333"><strong>as</strong></span> pool:

    future1 = pool.submit(action, <span style="color:#880000">1000</span>)

    <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>get_result</strong></span>(future):
        print(f<span style="color:#880000">"单个任务返回:{future.result()}"</span>)

    <span style="color:#888888"># 为future1添加线程完成的回调函数</span>
    future1.add_done_callback(get_result)

    print(<span style="color:#880000">'------------------------------'</span>)
    <span style="color:#888888"># 使用线程执行map计算</span>
    results = pool.map(action, (<span style="color:#880000">50</span>, <span style="color:#880000">100</span>, <span style="color:#880000">150</span>))
    <span style="color:#333333"><strong>for</strong></span> r <span style="color:#333333"><strong>in</strong></span> results:
        print(f<span style="color:#880000">"多个任务返回:{r}"</span>)
</span></span></span></span>

异步 IO/多进程/多线程对比

异步 IO(asyncio)、多进程(multiprocessing)、多线程(multithreading)

IO 密集型应用CPU等待IO时间远大于CPU 自身运行时间，太浪费；

常见的 IO 密集型业务包括：浏览器交互、磁盘请求、网络爬虫、数据库请求等

Python 世界对于 IO 密集型场景的并发提升有 3 种方法：多进程、多线程、异步 IO(asyncio);

理论上讲asyncio是性能最高的，原因如下：

进程、线程会有CPU上下文切换
进程、线程需要内核态和用户态的交互，性能开销大；而协程对内核透明的,只在用户态运行
进程、线程并不可以无限创建，最佳实践一般是 CPU*2；而协程并发能力强，并发上限理论上取决于操作系统IO多路复用(Linux下是 epoll)可注册的文件描述符的极限

那asyncio的实际表现是否如理论上那么强，到底强多少呢？我构建了如下测试场景：

访问500台 DB，并sleep 100ms模拟业务查询

方法 1；顺序串行一台台执行
方法 2：多进程
方法 3：多线程
方法 4：asyncio
方法 5：asyncio+uvloop

最后的和官方asyncio 最大不同是用 Cython+libuv 重新实现了asyncio 的事件循环(event loop)部分, asyncio+uvloop

官方测试性能是 node.js的 2 倍，持平 golang。

以下测试代码需要 Pyhton3.7+：

顺序串行一台台执行

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> records

user = <span style="color:#880000">"root"</span>
pwd = <span style="color:#880000">"123456"</span>
port = <span style="color:#880000">3306</span>
hosts = []  <span style="color:#888888"># 500台 db列表</span>

<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>query</strong></span>(host):
    conn = records.Database(
    f<span style="color:#880000">'mysql+pymysql://{user}:{pwd}@{host}:{port}/mysql?charset=utf8mb4'</span>)
    rows = conn.query(<span style="color:#880000">'select sleep(0.1);'</span>)
    print(rows[<span style="color:#880000">0</span>])

<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    <span style="color:#333333"><strong>for</strong></span> h <span style="color:#333333"><strong>in</strong></span> hosts:
        query(h)

<span style="color:#888888"># main entrance</span>
<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    main()
</span></span></span></span>

多进程

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>from</strong></span> concurrent <span style="color:#333333"><strong>import</strong></span> futures
<span style="color:#333333"><strong>import</strong></span> records

user = <span style="color:#880000">"root"</span>
pwd = <span style="color:#880000">"123456"</span>
port = <span style="color:#880000">3306</span>
hosts = []  <span style="color:#888888"># 500台 db列表</span>

<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>query</strong></span>(host):
    conn = records.Database(
    f<span style="color:#880000">'mysql+pymysql://{user}:{pwd}@{host}:{port}/mysql?charset=utf8mb4'</span>)
    rows = conn.query(<span style="color:#880000">'select sleep(0.1);'</span>)
    print(rows[<span style="color:#880000">0</span>])

<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    <span style="color:#333333"><strong>with</strong></span> futures.ProcessPoolExecutor() <span style="color:#333333"><strong>as</strong></span> executor:
        <span style="color:#333333"><strong>for</strong></span> future <span style="color:#333333"><strong>in</strong></span> executor.map(query,hosts):
            <span style="color:#333333"><strong>pass</strong></span>

<span style="color:#888888"># main entrance</span>
<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    main()
</span></span></span></span>

多线程

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>from</strong></span> concurrent <span style="color:#333333"><strong>import</strong></span> futures
<span style="color:#333333"><strong>import</strong></span> records

user = <span style="color:#880000">"root"</span>
pwd = <span style="color:#880000">"123456"</span>
port = <span style="color:#880000">3306</span>
hosts = []  <span style="color:#888888"># 500台 db列表</span>

<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>query</strong></span>(host):
    conn = records.Database(
    f<span style="color:#880000">'mysql+pymysql://{user}:{pwd}@{host}:{port}/mysql?charset=utf8mb4'</span>)
    rows = conn.query(<span style="color:#880000">'select sleep(0.1);'</span>)
    print(rows[<span style="color:#880000">0</span>])

<span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    <span style="color:#333333"><strong>with</strong></span>  .ThreadPoolExecutor() <span style="color:#333333"><strong>as</strong></span> executor:
        <span style="color:#333333"><strong>for</strong></span> future <span style="color:#333333"><strong>in</strong></span> executor.map(query,hosts):
            <span style="color:#333333"><strong>pass</strong></span>

<span style="color:#888888"># main entrance</span>
<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    main()
</span></span></span></span>

asyncio

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> asyncio
<span style="color:#333333"><strong>from</strong></span> databases <span style="color:#333333"><strong>import</strong></span> Database 

user = <span style="color:#880000">"root"</span>
pwd = <span style="color:#880000">"123456"</span>
port = <span style="color:#880000">3306</span>
hosts = []  <span style="color:#888888"># 500台 db列表</span>

<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>query</strong></span>(host):
    DATABASE_URL = f<span style="color:#880000">'mysql+pymysql://{user}:{pwd}@{host}:{port}/mysql?charset=utf8mb4'</span>
    <span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>with</strong></span> Database(DATABASE_URL) <span style="color:#333333"><strong>as</strong></span> database:
        query = <span style="color:#880000">'select sleep(0.1);'</span>
        rows = <span style="color:#333333"><strong>await</strong></span> database.fetch_all(query=query)
        print(rows[<span style="color:#880000">0</span>])

<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    tasks = [asyncio.create_task(query(host)) <span style="color:#333333"><strong>for</strong></span> host <span style="color:#333333"><strong>in</strong></span> hosts]
    <span style="color:#333333"><strong>await</strong></span> asyncio.gather(*tasks)

<span style="color:#888888"># main entrance</span>
<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    asyncio.run(main())
</span></span></span></span>

asyncio+uvloop

<span style="color:#333333"><span style="background-color:#fefefe"><span style="color:#444444"><span style="background-color:#f6f6f6"><span style="color:#333333"><strong>import</strong></span> asyncio
<span style="color:#333333"><strong>import</strong></span> uvloop
<span style="color:#333333"><strong>from</strong></span> databases <span style="color:#333333"><strong>import</strong></span> Database

user = <span style="color:#880000">"root"</span>
pwd = <span style="color:#880000">"123456"</span>
port = <span style="color:#880000">3306</span>
hosts = []  <span style="color:#888888"># 500台 db列表</span>

<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>query</strong></span>(host):
    DATABASE_URL = f<span style="color:#880000">'mysql+pymysql://{user}:{pwd}@{host}:{port}/mysql?charset=utf8mb4'</span>
    <span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>with</strong></span> Database(DATABASE_URL) <span style="color:#333333"><strong>as</strong></span> database:
        query = <span style="color:#880000">'select sleep(0.1);'</span>
        rows = <span style="color:#333333"><strong>await</strong></span> database.fetch_all(query=query)
        print(rows[<span style="color:#880000">0</span>])


<span style="color:#333333"><strong>async</strong></span> <span style="color:#333333"><strong>def</strong></span> <span style="color:#880000"><strong>main</strong></span>():
    tasks = [asyncio.create_task(query(host)) <span style="color:#333333"><strong>for</strong></span> host <span style="color:#333333"><strong>in</strong></span> hosts]
    <span style="color:#333333"><strong>await</strong></span> asyncio.gather(*tasks)

<span style="color:#888888"># main entrance</span>
<span style="color:#333333"><strong>if</strong></span> __name__ == <span style="color:#880000">'__main__'</span>:
    uvloop.install()
    asyncio.run(main())
</span></span></span></span>

运行时间对比