我正在尝试获取一个项目列表,并根据API的某些处理检查它们的状态更改。该列表将被手动填充,数量可能会变化到几千个。在
我试图编写一个脚本,使多个同时连接到API,以保持检查状态更改。对于每个项目,一旦状态更改,必须停止检查尝试。基于阅读其他关于Stackoverflow的文章(特别是What is the fastest way to send 100,000 HTTP requests in Python?),我得出了以下代码。但是脚本总是在处理完列表一次后停止。我做错什么了?在
我面临的另一个问题是keyboard interrup方法从不触发(我尝试使用Ctrl+C,但它不会杀死脚本)。在from urlparse import urlparse
from threading import Thread
import httplib, sys
from Queue import Queue
requestURLBase = "https://example.com/api"
apiKey = "123456"
concurrent = 200
keepTrying = 1
def doWork():
while keepTrying == 1:
url = q.get()
status, body, url = checkStatus(url)
checkResult(status, body, url)
q.task_done()
def checkStatus(ourl):
try:
url = urlparse(ourl)
conn = httplib.HTTPConnection(requestURLBase)
conn.request("GET", url.path)
res = conn.getresponse()
respBody = res.read()
conn.close()
return res.status, respBody, ourl #Status can be 210 for error or 300 for successful API response
except:
print "ErrorBlock"
print res.read()
conn.close()
return "error", "error", ourl
def checkResult(status, body, url):
if "unavailable" not in body:
print status, body, url
keepTrying = 1
else:
keepTrying = 0
q = Queue(concurrent * 2)
for i in range(concurrent):
t = Thread(target=doWork)
t.daemon = True
t.start()
try:
for value in open('valuelist.txt'):
fullUrl = requestURLBase + "?key=" + apiKey + "&value=" + value.strip() + "&years="
print fullUrl
q.put(fullUrl)
q.join()
except KeyboardInterrupt:
sys.exit(1)
我是Python新手,所以可能也有语法错误。。。我绝对不熟悉多线程,所以可能我也做错了别的事情。在