转载请注明出处https://blog.csdn.net/weixin_45163516
利用Beautiful模块和强大的正则表达式来爬取网页
from bs4 import BeautifulSoup
from urllib.request import urlopen
import re
import random
base_url = "https://baike.baidu.com"
his = ["/item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711"]
url = base_url + his[-1]
html = urlopen(url).read().decode('utf-8')
soup = BeautifulSoup(html, features='lxml')
print(soup.find('h1').get_text(), ' url: ', his[-1])
# find valid urls
sub_urls = soup.find_all("a", {"target": "_blank", "href": re.compile("/item/(%.{2})+$")})
if len(sub_urls) != 0:
his.append(random.sample(sub_urls, 1)[0]['href'])
else:
# no valid sub link found
his.pop()
print(his)
his = ["/item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711"]
for i in range(20):
url = base_url + his[-1]
html = urlopen(url).read().decode('utf-8')
soup = BeautifulSoup(html, features='lxml')
print(i, soup.find('h1').get_text(), ' url: ', his[-1])
# find valid urls
sub_urls = soup.find_all("a", {"target": "_blank", "href": re.compile("/item/(%.{2})+$")})
if len(sub_urls) != 0:
his.append(random.sample(sub_urls, 1)[0]['href'])
else:
# no valid sub link found
his.pop()
网络爬虫 url: /item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711
[’/item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711’, ‘/item/%E6%8E%92%E5%BA%8F%E7%AE%97%E6%B3%95’]
0 网络爬虫 url: /item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711
1 www url: /item/%E4%B8%87%E7%BB%B4%E7%BD%91
2 www url: /item/%E4%B8%87%E7%BB%B4%E7%BD%91
3 千年技术奖 url: /item/%E5%8D%83%E5%B9%B4%E6%8A%80%E6%9C%AF%E5%A5%96
4 京都大学 url: /item/%E4%BA%AC%E9%83%BD%E5%A4%A7%E5%AD%A6
5 京都大学 url: /item/%E4%BA%AC%E9%83%BD%E5%A4%A7%E5%AD%A6
6 义项 url: /item/%E4%B9%89%E9%A1%B9
7 祝福 url: /item/%E7%A5%9D%E7%A6%8F
8 南腔北调集 url: /item/%E5%8D%97%E8%85%94%E5%8C%97%E8%B0%83%E9%9B%86
9 祝福 url: /item/%E7%A5%9D%E7%A6%8F
10 鲁迅 url: /item/%E9%B2%81%E8%BF%85
11 魏晋风度及文章与药及酒之关系 url: /item/%E9%AD%8F%E6%99%8B%E9%A3%8E%E5%BA%A6%E5%8F%8A%E6%96%87%E7%AB%A0%E4%B8%8E%E8%8D%AF%E5%8F%8A%E9%85%92%E4%B9%8B%E5%85%B3%E7%B3%BB
当爬取到第十一个关键字之后出错了,这个是因为在该篇文章中没有了可以跳转的网页关键字,所以出现了错误。
学习自:莫烦python