一、分析观察网站结构
1.观察分析URL
2.查看项目名称对应元素
3.查看单价对应元素
4.查看户型等的对应元素
二、编写代码
操作过多会被锁定,就目前所学只尝试了一些简单代码,选取了鄞州区和宁海县第一页的项目名称、单价、户型等信息,宁波二手房信息获取方法如下:
from urllib.request import urlopen
from bs4 import BeautifulSoup
citys=["yinzhouqu2","ninghaixian"]
for i in citys:
html = urlopen("https://bj.lianjia.com/ershoufang/"+i)
soup = BeautifulSoup(html, "lxml")
name_list = soup.find_all("a", {"data-el": "region"})
price_list = soup.findAll("div", {"class": "unitPrice"})
info_list = soup.findAll("div", {"class": "houseInfo"})
for j in name_list:
print(j.get_text())
for p in price_list:
print(p.get_text())
for q in info_list:
print(q.get_text())
简单整理后,结果如下:
三、拓展
类似的方法可以还可以获取小说中有哪些人物,观察结果和代码如下:
from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://www.pythonscraping.com/pages/warandpeace.html")
soup = BeautifulSoup(html, "lxml")
nameList = soup.findAll("span", {"class":"green"})
for name in nameList:
print(name.get_text())
结果如下: