我要做一个事情,需要请求几万个网站,每个网站请求几百次,来获取网页状态码和html内容。
测试了下httplib的速度是urllib2的好几倍
1. [代码][Python]代码 跳至 [1] [全屏预览]
01 | import time |
02 |
03 | print time.time() |
04 | import urllib2 |
05 |
06 | response = urllib2.urlopen( 'http://www.cnseay.com/' ) |
07 |
08 | print response.getcode() |
09 | a = response.read() |
10 | response.close() |
11 |
12 |
13 | print time.time() |
14 | import httplib |
15 | conn = httplib.HTTPConnection( "www.cnseay.com" ) |
16 | conn.request( 'get' , '/' ) |
17 |
18 | res = conn.getresponse() |
19 |
20 | print res.status |
21 | b = res.read() |
22 | conn.close() |
23 | print time.time() |