Python爬虫例子1

网页

http://www.pythonscraping.com/pages/warandpeace.html

这是战争与和平，里面绿色的字是人名。

网页源代码

快捷键Ctrl+U即可显示网页源代码

观察源代码，发现都是<span class="red">或者<span class="green">

里面表示人名的绿色字体就是<span class="green">

Python抓取代码

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://www.pythonscraping.com/pages/warandpeace.html")
bsObj = BeautifulSoup(html, "lxml")
nameList = bsObj.findAll("span", {"class":"green"})

for name in nameList:
    print(name.get_text())

输出：

Anna
Pavlovna Scherer
Empress Marya
Fedorovna
Prince Vasili Kuragin
Anna Pavlovna
St. Petersburg
the prince
Anna Pavlovna
Anna Pavlovna
the prince
the prince
the prince
Prince Vasili
Anna Pavlovna
Anna Pavlovna
the prince
Wintzingerode
King of Prussia
le Vicomte de Mortemart
Montmorencys
Rohans
Abbe Morio
the Emperor
the prince
Prince Vasili
Dowager Empress Marya Fedorovna
the baron
Anna Pavlovna
the Empress
the Empress
Anna Pavlovna's
Her Majesty
Baron
Funke
The prince
Anna
Pavlovna
the Empress
The prince
Anatole
the prince
The prince
Anna
Pavlovna
Anna Pavlovna

代码来自《Python网络数据采集》一书