python 使用协程asyncio加run_in_executor线程池处理同时下载多文件

水星灭绝

于 2020-11-01 20:09:16 发布

阅读量4.1k

点赞数 1

分类专栏： python

本文链接：https://blog.csdn.net/wulong710/article/details/109435557

版权

本文介绍了如何使用Python的asyncio库结合run_in_executor线程池实现多文件的并发下载。通过这种方式，可以有效地避免阻塞，提高下载效率。

摘要由CSDN通过智能技术生成

设置默认线程池

from concurrent.futures import ThreadPoolExecutor

exe = ThreadPoolExecutor(2)
asyncio.get_event_loop().set_default_executor(exe)

同时下载多文件，碰到阻塞的函数则用run_in_executor来新开线程跳过阻塞

# coding=utf-8

import asyncio
import os
import requests


async def downOne(loop, url):
    name = os.path.basename(url)
    print("downOne:start {}".format(name))
    # loop = asyncio.get_event_loop()
    # requests.get(url)
    result = await loop.run_in_executor(None, requests.get, url)
    print("downOne:middle {}".format(name))
    if result.status_code == 200:
        with open(name, "wb") as fp:
            # fp.write(result.content)
            r = await loop.run_in_executor(None, fp.write, result.content)
            print("downOne: r = {}".format(r))
    print("downOne:endxx {}".format(name))
    return "