爬虫随机爬取百度百科"网络爬虫"

转载请注明出处https://blog.csdn.net/weixin_45163516

利用Beautiful模块和强大的正则表达式来爬取网页

from bs4 import BeautifulSoup
from urllib.request import urlopen
import re
import random


base_url = "https://baike.baidu.com"
his = ["/item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711"]


url = base_url + his[-1]

html = urlopen(url).read().decode('utf-8')
soup = BeautifulSoup(html, features='lxml')
print(soup.find('h1').get_text(), '    url: ', his[-1])

# find valid urls
sub_urls = soup.find_all("a", {"target": "_blank", "href": re.compile("/item/(%.{2})+$")})

if len(sub_urls) != 0:
    his.append(random.sample(sub_urls, 1)[0]['href'])
else:
    # no valid sub link found
    his.pop()
print(his)

his = ["/item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711"]

for i in range(20):
    url = base_url + his[-1]

    html = urlopen(url).read().decode('utf-8')
    soup = BeautifulSoup(html, features='lxml')
    print(i, soup.find('h1').get_text(), '    url: ', his[-1])

    # find valid urls
    sub_urls = soup.find_all("a", {"target": "_blank", "href": re.compile("/item/(%.{2})+$")})

    if len(sub_urls) != 0:
        his.append(random.sample(sub_urls, 1)[0]['href'])
    else:
        # no valid sub link found
        his.pop()

网络爬虫 url: /item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711
[’/item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711’, ‘/item/%E6%8E%92%E5%BA%8F%E7%AE%97%E6%B3%95’]
0 网络爬虫 url: /item/%E7%BD%91%E7%BB%9C%E7%88%AC%E8%99%AB/5162711
1 www url: /item/%E4%B8%87%E7%BB%B4%E7%BD%91
2 www url: /item/%E4%B8%87%E7%BB%B4%E7%BD%91
3 千年技术奖 url: /item/%E5%8D%83%E5%B9%B4%E6%8A%80%E6%9C%AF%E5%A5%96
4 京都大学 url: /item/%E4%BA%AC%E9%83%BD%E5%A4%A7%E5%AD%A6
5 京都大学 url: /item/%E4%BA%AC%E9%83%BD%E5%A4%A7%E5%AD%A6
6 义项 url: /item/%E4%B9%89%E9%A1%B9
7 祝福 url: /item/%E7%A5%9D%E7%A6%8F
8 南腔北调集 url: /item/%E5%8D%97%E8%85%94%E5%8C%97%E8%B0%83%E9%9B%86
9 祝福 url: /item/%E7%A5%9D%E7%A6%8F
10 鲁迅 url: /item/%E9%B2%81%E8%BF%85
11 魏晋风度及文章与药及酒之关系 url: /item/%E9%AD%8F%E6%99%8B%E9%A3%8E%E5%BA%A6%E5%8F%8A%E6%96%87%E7%AB%A0%E4%B8%8E%E8%8D%AF%E5%8F%8A%E9%85%92%E4%B9%8B%E5%85%B3%E7%B3%BB

当爬取到第十一个关键字之后出错了，这个是因为在该篇文章中没有了可以跳转的网页关键字，所以出现了错误。

学习自:莫烦python