上篇写了一个单进程爬224个页面需要用450秒的时间,这次用下python的多进程的方式去改写一下,尽管大家都说python的多进程是鸡肋,但存在就是合理,多进程思路也很简单。
1,创建多个Queue队列,在循环links链接时将每个link链接put进queue列表里。
2.开启多个线程同时将每个queue join进每个线程,当线程暂用时会进入阻塞状态
3.当queue队列长度为0时,向进程发送NONE通知线程关闭。
import time
import requests
from lxml import etree
import threading
from queue import Queue
start_url = "http://www.qianmu.org/2019USNEWS%E4%B8%96%E7%95%8C%E5%A4%A7%E5%AD%A6%E6%8E%92%E5%90%8D%E7%BE%8E%E5%9B%BD%E5%9B%BD%E5%86%85%E6%8E%92%E5%90%8D"
download_pages = 0
cookies = {
"JSESSIONID": "61C0EDC5303A67FEDA043CCDC419A8D6"
}
link_queue = Queue()
threads_num = 50
threads = []
def fetch(url):
"""请求并下载网页"""
print("正在请求%s" % url)
r = requests.get(url, cookies=cookies)
text = r.text
print("请求结果%s" % r.reason)
if r.status_code != 200:
r.raise_for_status()
global download_pages
download_pages += 1
return text.replace('\t', '')
def parse_university(url):
time.sleep(1)
selector = etree.HTML(fetch(url))
# print(dir(selector))
data = {}
data['name'] = selector.xpath('//div[@id="wikiContent"]/h1/text()')[0]
table = selector.xpath('//div[@id="wikiContent"]/div[@class="infobox"]/table')
if table:
table = table[0]
keys = table.xpath('.//td[1]/p/text()')
cols = table.xpath('.//td[2]')
values = [' '.join(col.xpath('.//text()')) for col in cols]
if len(keys) != len(values):
return None
data.update(zip(keys, values))
return data
def process_data(data):
if data:
print(data)
def download():
while True:
# 阻塞直到从队列里获取一条消息
link = link_queue.get()
if link is None:
break
data = parse_university(link)
process_data(data)
link_queue.task_done()
print('还有%s个队列' % link_queue.qsize())
if __name__ == '__main__':
# 记录程序启动时间
start_time = time.time()
# 1.请求入口页面
selector = etree.HTML(fetch(start_url))
# 2.提取页面a标签链接
links = selector.xpath('//div[@id="content"]//tr[position()>1]/td[2]/a/@href')
for link in links:
# 将链接放入列表中。
link_queue.put(link)
# 启动线程,并将线程对象放入一个列表保存
for i in range(threads_num):
t = threading.Thread(target=download)
t.start()
threads.append(t)
# 阻塞队列直到队列被清空
link_queue.join()
# 向队列发送N个NONE,通知线程退出
for i in range(threads_num):
link_queue.put(None)
# 退出线程
for t in threads:
t.join()
cost_time = time.time() - start_time
print("dowmlooad %d pages, cost %.2f second" % (download_pages, cost_time))
主要修改
和添加download方法
实验结果:
单进程用了400多秒,开启50个线程90多秒。python多进程还是有点效果。