如果没有更多关于您正在做什么的信息,很难说是肯定的,但是一个简单的线程方法可能是有意义的。
假设您有一个处理单个ID的简单函数:import requests
url_t = "http://localhost:8000/records/%i"
def process_id(id):
"""process a single ID"""
# fetch the data
r = requests.get(url_t % id)
# parse the JSON reply
data = r.json()
# and update some data with PUT
requests.put(url_t % id, data=data)
return data
您可以将其扩展为处理一系列ID的简单函数:def process_range(id_range, store=None):
"""process a number of ids, storing the results in a dict"""
if store is None:
store = {}
for id in id_range:
store[id] = process_id(id)
return store
最后,您可以很容易地将子范围映射到线程上,以允许一定数量的请求是并发的:from threading import Thread
def threaded_process_range(nthreads, id_range):
"""process the id range in a specified number of threads"""
store = {}
threads = []
# create the threads
for i in range(nthreads):
ids = id_range[i::nthreads]
t = Thread(target=process_range, args=(ids,store))
threads.append(t)
# start the threads
[ t.start() for t in threads ]
# wait for the threads to finish
[ t.join() for t in threads ]
return store
如果您的单个任务占用的时间范围更广,您可能希望使用ThreadPool,它将一次分配一个作业(如果单个任务非常小,通常会较慢,但在异构情况下可以保证更好的平衡)。