之前写的代码都是串行的单线程爬虫,当爬取页面数量更多,数据量更大时,速度明显降低,这里使用Python Multiprocessing库的进程池方法测试多进程爬虫的效率,爬取糗事百科文字板块的用户ID,发表段子的文字信息、好笑数量和评论数量这几个数据,由于只是测试性能,对爬取的数据不进行保存。上代码:
import requests
import re
import time
from multiprocessing import Pool
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36'
}
def re_scraper(url):
res = requests.get(url,headers=headers)
ids = re.findall('<h2>(.*?)</h2>', res.text, re.S)
contents = re.findall('<div class="content"><span>(.*?)</span></div>', res.text, re.S)
laughts = re.findall('<span class="stats-vote"><i class="number">(\d+)</i>', res.text, re.S)
comments = re.findall('<i class="number">(\d+)</i>评论', res.text, re.S)
for id,content,laught,comment in zip(ids,contents,laughts,comments):
info = {
'id':id,
'content':content,
'laught':laught,
'comment':comment
}
#只爬取,不存储,测试性能
return
if __name__=='__main__':
urls = ['https://www.qiushibaike.com/text/page/{}/'.format(str(i)) for i in range(1,36)]
start_1 = time.time()
for url in urls:
re_scraper(url)
end_1 = time.time()
print("串行爬虫耗费时间:",end_1-start_1)
start_2 = time.time()
pool = Pool(processes=2)
pool.map(re_scraper,urls)
end_2 = time.time()
print("两个进程爬虫耗费时间:",end_2-start_2)
start_3 = time.time()
pool = Pool(processes=4)
pool.map(re_scraper,urls)
end_3 = time.time()
print("四个进程爬虫耗费时间:", end_3 - start_3)
结果如下:
串行爬虫耗费时间: 21.60699963569641
两个进程爬虫耗费时间: 11.561000347137451
四个进程爬虫耗费时间: 5.212999820709228
可以看出多进程爬虫确实效率提升明显