完成页面爬取后需要对返回的HTML字符串(或者列表)进行处理及解析数据
BeautifulSoup4安装方式
pip install beautifulsoup4
pip install lxml
pip install html5lib
引入方式:
from bs4 import BeautifulSoup
读取并定位数据示例,以360搜索页面为例
# -*- coding: utf-8 -*-
from urllib import request
import ssl
from bs4 import BeautifulSoup
ssl._create_default_https_context = ssl._create_unverified_context
url = 'https://www.so.com/?src=pclm&ls=safarimac'
headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) \
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36'}
req = request.Request(url, headers=headers)
response = request.urlopen(req)
html = response.read()
bs = BeautifulSoup(html, "html.parser", from_encoding="utf8")
# print(bs.nav) # 定位到nav标签内容
nav = bs.nav
a_links = nav.find_all("a")
# print(a_links)
links = []
for link in a_links:
links.append(link.get("href"))
for i in links:
print(i)
find_all方法可以在文本中去查找对应标签。如上代码就是查找了所有的a标签
要定位子节点方式为
父.子