pycurl同步变异步示例

最新推荐文章于 2024-05-03 11:07:56 发布

铁皮石斛hf

最新推荐文章于 2024-05-03 11:07:56 发布

阅读量236

点赞数 1

分类专栏： python 文章标签： python linux 开发语言

本文链接：https://blog.csdn.net/sinat_33353575/article/details/130168079

版权

python 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

文章目录

centos7 python 调用 pycurl 报错
- 环境信息
- 现象如下
解决
将pycurl变成异步的脚本
参考

centos7 python 调用 pycurl 报错

朋友分享了一个将pycurl变成异步的示例脚本，于是试一下，没想到pycurl还报错了。

环境信息

system: CentOS 7.9
python: Python 3.9.12
pycurl: pycurl-7.45.2

现象如下

# /data/apps/opt/miniconda3/bin/python -m venv /opt/pycurl

# /opt/pycurl/bin/pip install pycurl
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting pycurl
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a8/af/24d3acfa76b867dbd8f1166853c18eefc890fc5da03a48672b38ea77ddae/pycurl-7.45.2.tar.gz (234 kB)
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for pycurl, since package 'wheel' is not installed.
Installing collected packages: pycurl
  Running setup.py install for pycurl ... done
Successfully installed pycurl-7.45.2

调用报错

# /opt/pycurl/bin/python /tmp/test_curl.py 
Traceback (most recent call last):
  File "/tmp/test_curl.py", line 2, in <module>
    import pycurl
ImportError: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (none/other)

解决

参照官方文档

# 非必须，但是还是升一下
# /opt/pycurl/bin/pip install --upgrade pip

# 卸载已经安装的pycurl
# /opt/pycurl/bin/pip  uninstall pycurl

# 声明环境变量，使用nss, 当然需要先安装 nss-devel 的包
# export PYCURL_SSL_LIBRARY=nss

# 为install添加compile参数
# /opt/pycurl/bin/pip install --compile pycurl
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting pycurl
  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a8/af/24d3acfa76b867dbd8f1166853c18eefc890fc5da03a48672b38ea77ddae/pycurl-7.45.2.tar.gz (234 kB)
  Preparing metadata (setup.py) ... done
Installing collected packages: pycurl
  DEPRECATION: pycurl is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for pycurl ... done
Successfully installed pycurl-7.45.2

再次测试, 成功运行

# /opt/pycurl/bin/python /tmp/test_curl.py 
http://example.com start 1681530336.2565134
http://example.org start 1681530336.26023
http://example.net start 1681530336.2611377
http://example.com end 1681530336.5996385
http://example.org end 1681530336.601178
http://example.net end 1681530336.6026258
Request 1:
URL: http://example.com

将pycurl变成异步的脚本

重点就是asyncio.to_thread()函数。

import asyncio
import pycurl
import time

# 定义一个异步函数，用于进行 HTTP 请求
async def fetch(url):
    # 创建 Curl 对象
    c = pycurl.Curl()
    # 设置 URL
    c.setopt(pycurl.URL, url)
    # 设置回调函数，用于处理 HTTP 请求结果
    buffer = bytearray()
    c.setopt(pycurl.WRITEFUNCTION, buffer.extend)
    # 记录开始时间
    start_time = time.time()
    # 执行 HTTP 请求
    print(url, "start", start_time)
    await asyncio.to_thread(c.perform)
    # 记录结束时间
    end_time = time.time()
    print(url, "end", end_time)
    # 关闭 Curl 对象
    c.close()
    # 返回请求结果和时长
    return buffer.decode('utf-8'), end_time - start_time

async def main():
    # 定义需要请求的 URL 列表
    urls = ['http://example.com', 'http://example.org', 'http://example.net']
    # 创建任务列表
    tasks = []
    # 遍历 URL 列表，创建异步任务
    for url in urls:
        tasks.append(fetch(url))
    # 执行异步任务
    results = await asyncio.gather(*tasks)
    # 输出结果
    for i, (content, duration) in enumerate(results, 1):
        print(f"Request {i}:")
        print(f"URL: {urls[i-1]}")
        print(f"Content: {content[:100]}...")
        print(f"Duration: {duration:.2f} seconds")
        print()
# 运行主函数
asyncio.run(main())