1.背景
爬虫模拟浏览器获取网页的数据。
网页有唯一的URL地址,页面信息由HTML来描述,并使用HTTP/HTTPS协议来传输HTML网页数据;
关于:HTML(HyperText Markup Language)超文本标记语言是一种用于创建网页的标准标记语言。
2.安装
pip install bs4
C:\Users\G7>python -V
Python 3.6.8
C:\Users\G7>pip install bs4
Collecting bs4
Downloading bs4-0.0.1.tar.gz (1.1 kB)
Collecting beautifulsoup4
Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
|████████████████████████████████| 115 kB 285 kB/s
Collecting soupsieve>1.2; python_version >= "3.0"
Downloading soupsieve-2.0.1-py3-none-any.whl (32 kB)
Building wheels for collected packages: bs4
Building wheel for bs4 (setup.py) ... done
Created wheel for bs4: filename=bs4-0.0.1-py3-none-any.whl size=1277 sha256=5fd8e30825a0170b1f16f78e84fba037685af463843685dae9c0593bc7a8c926
Stored in directory: c:\users\g7\appdata\local\pip\cache\wheels\19\f5\6d\a97dd4f22376d4472d5f4c76c7646876052ff3166b3cf71050
Successfully built bs4
Installing collected packages: soupsieve, beautifulsoup4, bs4
Successfully installed beautifulsoup4-4.9.3 bs4-0.0.1 soupsieve-2.0.1
3.简单示例
import requests
import bs4
if __name__=="__main__":
url="http://www.cwl.gov.cn/"
#构造一个向服务器请求资源的url对象
req=requests.get(url=url)
#设置请求网页的编码格式
req.encoding="utf-8"
#获得网页的源代码
html=req.text
#html parser解析器
#BeautifulSoup 是一个可以从HTML或XML文件中提取数据的Python库
bf=bs4.BeautifulSoup(html, "lxml")
#遍历文档进行搜索
img_url=bf.find_all(class_="n_item")[0]
print(img_url)
#<div class="n_item"><a href="/zzjg/" target="_blank">组织机构</a></div>