Beautiful Soup库是解析HTML页面信息标记与提取方法,解析、维护、遍历“标签树”的功能库。
初步使用Beautiful Soup库
from bs4 import BeautifuSoup
#html为指定要解析的网页
soup = BeautifulSoup(<p>html</p>, "html.parser")
运行的结果是HTML页面的标签
Beautiful Soup库解析器
解析器 | 使用方法 | 条件 |
---|---|---|
bs4的HTML解析器 | BeautifulSoup(mk,“html.parser”) | 安装bs4库 |
lxml的HTML解析器 | BeautifulSoup(mk,“lxml”) | pip install lxml |
lxml的XML解析器 | BeautifulSoup(mk,“xml”) | pip install lxml |
html5lib的解析器 | BeautifulSoup(mk,“html5lib”) | pip install html5lib |
Beautiful Soup类 的基本元素
基本元素 | 说明 |
---|---|
Tag | 标签,最基本的组织单元,分为用<>和</>标明开头和结尾 |
Name | 标签的名字,< p>…< /p>的名字是“p”,格式:< tag>.name |
Attributes | 标签的属性,字典形式组织,格式:< tag>.attrs |
NavigableString | 标签内非属性字符串,<>…</>中字符串,格式:< tag>.string |
Comment | 标签内字符串的注释部分,一种特殊的Comment类型 |
Beautiful Soup类 的基本元素使用
利用requests获取页面信息
import requests
r = requests.get("https://python123.io/ws/demo.html")
demo = r.text
利用BeautifulSoup类进行页面解析
from bs4 import BeautifuSoup
soup = BeautifulSoup(demo, "html.parser")
#获取解析的内容
print(soup.prettify())
结果:
<html>
<head>
<title>
This is a python demo page
</title>
</head>
<body>
<p class="title">
<b>
The demo python introduces several python courses.
</b>
</p>
<p class="course">
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
Basic Python
</a>
and
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
Advanced Python
</a>
.
</p>
</body>
</html>
获取浏览器头部
soup.title
结果为:
<title>This is a python demo page</title>
获取a标签内容
tag = soup.a
tag
结果为:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
name元素
获取a标签的父标签的名字和父标签的父标签的名字(返回的类型是字符串)
soup.a.parent.name
结果为:
'p'
soup.a.parent.parent.name
结果为:
‘body’
Attributes元素
标签的属性是字典
获取a标签的属性
soup.a.attrs
结果为:
{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
a的class属性
soup.a.attrs['class']
结果为:
['py1']
a的链接属性
soup.a.attrs['href']
结果为:
'http://www.icourse163.org/course/BIT-268001'
标签类型
标签的类型是标签,在bs4中定义为特殊类型
type(soup.a)
结果为:
<class 'bs4.element.Tag'>
字符串信息
soup.a.string
结果为:
'Basic Python'
#获取p标签内容
soup.p
<p class="title"><b>The demo python introduces several python courses.</b></p>
#获取p标签的字符串信息
soup.p.string
'The demo python introduces several python courses.'
#p标签的字符串类型
字符串类型
bs4中的字符串类型定义为NavigableString
type(soup.p.string)
结果:
<class 'bs4.element.NavigableString'>
Comment基础元素
注释类型,以**<!–内容-- >**为注释
newsoup = BeautifulSoup("<b><!--This is a comment--></b><p>This is not a comment</p>","html.parser")
#获取b标签下的字符串
newsoup.b.string
'This is a comment'
#获取b标签内容
newsoup.b
<b><!--This is a comment--></b>
#获取注释类型
type(newsoup.b.string)
<class 'bs4.element.Comment'>
#获取p标签下的字符串
newsoup.p.string
'This is not a comment'
#获取字符串类型
type(newsoup.p.string)
<class 'bs4.element.NavigableString'>
BeautifulSoup标签树的遍历
与HTML遍历结构相同,标签树也存在上行、下行和平行遍历
标签树的下行遍历
属性 | 说明 |
---|---|
.contents | 子节点的列表,将< tag>所有儿子节点存入列表 |
.children | 子节点的迭代类型,与.contents类似,用于循环遍历儿子节点 |
.descendants | 子孙节点的迭代类型,包含所有子孙节点,用于循环遍历 |
.contents
soup.head
<head><title>This is a python demo page</title></head>
soup.head.contents
[<title>This is a python demo page</title>]
soup.body.contents
['\n', <p class="title"><b>The demo python introduces several python courses.</b></p>, '\n', <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>, '\n']
len(soup.body.contents)
5
soup.body.contents[1]
<p class="title"><b>The demo python introduces several python courses.</b></p>
.children
for child in soup.body.children:
print(child)
.descendants
for child in soup.body.descendants:
print(child)
标签树的上行遍历
属性 | 说明 |
---|---|
.parent | 节点的父亲标签 |
.parents | 杰斯按先辈标签的迭代类型,用于循环遍历先辈节点 |
soup.title.parent
<head><title>This is a python demo page</title></head>
soup.html.parent
<html><head><title>This is a python demo page</title></head>
<body>
<p class="title"><b>The demo python introduces several python courses.</b></p>
<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>
</body></html>
soup.parent
标签树的平行遍历
平行遍历必须发生在同一个父节点下的各子节点
属性 | 说明 |
---|---|
.next_sibling | 返回按照HTML文本顺序的下一个平行节点标签 |
.previous_sibling | 返回按照HTML文本顺序的上一个平行节点标签 |
.next_siblings | 迭代类型,返回按照HTML文本顺序的后续所有平行节点标签 |
.previous_siblings | 迭代类型,返回按照HTML文本顺序的谦虚所有平行节点标签 |
soup.a.next_sibling
' and '
soup.a.next_sibling.next_sibling
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
soup.a.previous_sibling
'Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\n'
soup.a.previous_sibling.previous_sibling
bs4库的编码
soup = BeautifulSoup("<p>中文</p>","html.parser")
soup.p.string
’中文‘
print(soup.p.prettify())
<p>
中文
</p>
总结:
format输出