案例
为了实现并发上传,我们可以使用concurrent.futures
模块中的ThreadPoolExecutor
来创建一个线程池并发地执行请求。以下是修改后的代码,将会并发地执行40次上传,每次最多10个并发:
import requests
import json
from concurrent.futures import ThreadPoolExecutor, as_completed
def send_request(session, url, headers, payload):
response = session.post(url, headers=headers, data=payload)
return response.text
# 参数配置
url = "http://127.0.0.1:8080/post"
payload = json.dumps({
"data": "content"
})
headers = {
'Content-Type': 'application/json'
}
# 创建一个会话以复用连接
session = requests.Session()
# 设置并发数和上传次数
concurrency_level = 10
upload_times = 40
# 使用ThreadPoolExecutor创建线程池
with ThreadPoolExecutor(max_workers=concurrency_level) as executor:
# 创建Future对象列表
futures_list = [
executor.submit(send_request, session, url, headers, payload)
for _ in range(upload_times)
]
# 等待Future对象完成并获取结果
for future in as_completed(futures_list):
try:
response_text = future.result()
print(response_text)
except Exception as e:
print(f"Request failed: {e}")
# 关闭会话
session.close()
在这个例子中,我们首先创建了一个requests.Session
对象,这是为了能够在多个请求之间复用连接,提高效率。然后,我们使用ThreadPoolExecutor
来创建一个线程池,最大并发数设置为10。通过executor.submit
方法,我们提交了40次任务到线程池中。
as_completed
函数用来迭代完成的任务(Future对象),然后我们通过调用result()
方法获取每个请求的返回内容。
最后,我们关闭了会话,以确保释放资源。在实际应用中,可能还需要根据服务器的响应和具体的API行为做一些额外的错误处理。
代码封装优化
为了提高代码的复用性并支持GET请求,我们可以定义一个类来管理HTTP请求,并提供发送GET和POST请求的方法。以下是一个简单的示例:
import requests
import json
from concurrent.futures import ThreadPoolExecutor, as_completed
class HTTPClient:
def __init__(self, concurrency_level=10):
self.session = requests.Session()
self.concurrency_level = concurrency_level
def send_get(self, url, headers={}, silence=False):
response = self.session.get(url, headers=headers)
self.handle_response(response, silence)
return response.text
def send_post(self, url, headers, payload, silence=False):
response = self.session.post(url, headers=headers, data=payload)
self.handle_response(response, silence)
return response.text
def send_requests_concurrently(self, method, url, headers, payload, times):
with ThreadPoolExecutor(max_workers=self.concurrency_level) as executor:
futures_list = [
executor.submit(method, url, headers, payload)
for _ in range(times)
]
responses = []
for future in as_completed(futures_list):
try:
response_text = future.result()
responses.append(response_text)
except Exception as e:
print(f"Request failed: {e}")
return responses
def handle_response(self, response, silence=False):
if silence:
return
# print(f"Headers: {response.headers}")
print(f"Body: {response.text}")
def close(self):
self.session.close()
# 使用HTTPClient类
if __name__ == "__main__":
# 实例化HTTPClient
client = HTTPClient(concurrency_level=10)
# 设置POST请求参数
url = "http://127.0.0.1:8080/post"
payload = json.dumps({
"data": "content"
})
headers = {
'Content-Type': 'application/json'
}
# 发送并发POST请求
responses = client.send_requests_concurrently(client.send_post, url, headers, payload, 40)
# for response_text in responses:
# print(response_text)
# 示例:发送并发GET请求
# responses = client.send_requests_concurrently(client.send_get, "http://example.com", {}, None, 40)
# for response_text in responses:
# print(response_text)
# 关闭客户端
client.close()
在这个封装中,我们创建了一个名为HTTPClient
的类,它负责管理HTTP会话和并发请求。send_get
和send_post
方法分别用于发送GET和POST请求。send_requests_concurrently
方法接受一个请求方法(如send_get
或send_post
)、URL、请求头、负载和请求次数作为参数,然后并发执行这些请求。
当需要发送请求时,只需创建一个HTTPClient
实例并调用相应的方法即可。这样的封装使代码更加整洁、可维护,并且可以很容易地扩展以支持其他类型的HTTP请求。在最后的使用示例中,我们发送了40次并发POST请求,也给出了并发GET请求的示例注释。