python request与grequests该如何选择

测试大大怪

于 2024-11-07 14:50:44 发布

阅读量1k

点赞数 14

文章标签： python 单元测试功能测试测试用例 appium 测试工具 postman

本文链接：https://blog.csdn.net/kk_lzvvkpj/article/details/143594053

版权

requests 和 grequests 是Python中用于发送HTTP请求的不同库。requests 是一个同步、阻塞式库，而 grequests 是基于 requests 封装的异步非阻塞库，它利用了 gevent 库提供的协程机制，能够并发发送多个请求。

选择哪个库取决于你的需求：

当你只需要顺序发送少量请求，或者不关心请求之间并发性能的时候，requests 是一个很好的选择，因为它易于理解和使用。

当你需要并发处理大量HTTP请求，提高整体性能，尤其是网络IO密集型任务时，如爬虫抓取大量页面、API服务端并发请求下游服务等场景下，grequests 更合适。‍

import requests
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
# 单线程同步请求
responses = [requests.get(url) for url in urls]
# 处理响应
for response in responses:
print(response.text)

import requests
import threading
# 创建一个处理请求的函数
def fetch_page(url):
response = requests.get(url)
return response.text
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
threads = []
# 启动多线程请求
for url in urls:
thread = threading.Thread(target=fetch_page, args=(url,))
threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
# 假设你有一个列表存储结果
results = [result for result in thread_results]

异步并发请求（使用grequests）

import grequests
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
# 异步并发请求
rs = (grequests.get(url) for url in urls)
responses = grequests.map(rs)
# 处理响应
for response in responses:
print(response.text)

超时控制（grequests可以更方便地处理超时）

import grequests
# 对每个请求设置超时时间为5秒
urls = [("https://example.com/page{}".format(i), {'timeout': 5}) for i in range(1, 4)]
rs = (grequests.get(url, **kwargs) for url, kwargs in urls)
responses = grequests.map(rs)
for response in responses:
if response is not None:
print(response.text)
else:
print(f"Request timed out for {response.request.url}")

带异常处理的异步请求（grequests可以在一个循环中统一处理异常）

import grequests
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
rs = [grequests.get(url) for url in urls]
def handle_exception(request, exception):
print(f"Error on URL {request.url}: {exception}")
responses = grequests.map(rs, size=10, exception_handler=handle_exception)
for response in responses:
if response is not None:
print(f"{response.request.url} - Response: {response.text}")