本文介绍了Python库Beautiful Soup的使用，包括安装、解析HTML、遍历标签树（如soup.<tag>、.contents、.children、.descendants、.parent等）、搜索标签（如find()、find_all()、CSS选择器）以及提取信息（如.string、.strings、.stripped_strings、获取属性值和节点名称）。通过示例展示了如何高效地解析和提取网页数据。
目录
1.2 标签树的下行遍历 .contents 和 .children 及.descendants
1.4 标签树的平行遍历 ： .next_sibling 和 .previous_sibling及.next_siblings 和 .previous_siblings
1.5 提取信息：.string、.strings、.stripped_strings
一、安装Beautiful soup
这篇文章也是受了崔庆才大神的启发，然后自己按照自己的思路梳理出来的，所以重复的内容这里就不赘述了。可以参考崔庆才的网站。这里放上链接：https://cuiqingcai.com/5548.html
二、解析并获取网页中想要的内容
Beautiful Soup是一个集解析和提取数据为一体的强大的库（它支持python标准库中的HTML解析器，还支持一些第三方解析器（lxml,html5li）)经BeautifulSoup解析后，会将复杂HTML文档转换成一个复杂的树形结构,每个标签都对应一个节点，每个节点都是python对象,并且BeautifulSoup提供了常用的遍历，查找，修改解析树的方法,可以从HTML或XML文件中提取数据。
先来看python支持的解析器的对比：
| 解析器 | 使用方法 | 优势 | 劣势 |
|---|---|---|---|
| Python标准库 html.parser | BeautifulSoup(markup, "html.parser") |
|
|
| lxml HTML 解析器 需要安装 lxml库 |
BeautifulSoup(markup, "lxml") |
|
|
| lxml XML 解析器 需要安装 lxml库 |
|
|
|
| html5lib 需要安装html5lib库 |
BeautifulSoup(markup, "html5lib") |
|
|
推荐使用lxml作为解析器,因为效率更高. 在Python2.7.3之前的版本和Python3中3.2.2之前的版本,必须安装lxml或html5lib, 因为那些Python版本的标准库中内置的HTML解析方法不够稳定.
那经BeautifulSoup解析后的所有对象可以归纳为4种:
-
Tag（Tagl对象当中有name属性和attrs属性）
-
NavigableString
-
BeautifulSoup
-
Comment
即：
有HTML基础的人都知道，HTML中最重要的就是标签，不同标签有不同的作用，标签当中有属性和元素内容，因此我们就从标签入手，一步一步分析beautiful soup是怎么提取HTML数据的。
1、标签树的遍历
1.1 soup.<tag>
想要获取标签及其内容，最简单的方法就是告诉它你想获取的tag的name.任何存在于HTML语法中的标签都可以用soup.<tag>访问获得，当HTML文档中存在多个相同<tag>对应内容时，soup.<tag>返回第一个。
比如：
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<div cless='out_wrapper'>
<div>
<h1>The Dormouse's story</h1>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
</div>
</div>
</body>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.title)
print(soup.head)
print(soup.p)运行结果如下：
<title>The Dormouse's story</title>
<div class="out_wrapper">
<div>
<h1>The Dormouse's story</h1>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
</div>
</div>
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.
</p>来看结果：首先打印输出 title 节点加里面的文字内容，接下来，我们又尝试选择了 div节点，结果返回的也是节点加其内部的所有内容,并且注意虽然虽然HTML网页中有两个div但是结果只输出了最外层也就是第一个找到的div。最后，选择了 p 节点。此时需要注意输出的结果只有第一个 p 节点及其内容，后面的几个 p 节点并没有选到。也就是说，当有多个节点的时候，这种选择方式只会输出第一个匹配到的节点，其他的内部或者后边的节点都不会输出。
1.2 标签树的下行遍历 .contents 和 .children 及.descendants
Beautiful Soup除了可以根据标签的名字返回标签及其内容之外，还提供了很多属性供我们遍历标签树，比如 .contents和.chaldren，.descendants都是tag对象的属性，其中.contents属性可以将tag的直接子节点以列表的方式输出；.children属性和.descendants返回的是生成器。.children返回的是tag标签直接后代的一个生成器，.descendants返回的是tag标签全部后代的生成器
例如：
html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<div class='out_wrapper'>
<div>
<h1>The Dormouse's story</h1>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
</div>
</div>
</body>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#实例化
print(soup.p.contents)
print('------'*10)
print(soup.p.children)
for i,child in enumerate(soup.p.children):
print(i,child)
['Once upon a time there were three little sisters; and their names were\n
', <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>, ',\n ', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ' and\n ', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, ';\n
and they lived at the bottom of a well.\n ']
------------------------------------------------------------
<list_iterator object at 0x000001FDBC52A430>
0 Once upon a time there were three little sisters; and their names were
1 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
2 ,
3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4 and
5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6 ;
and they lived at the bottom of a well. 根据结果可以看到：当我们要获取p的子节点的时候，结果以列表的形式返回了p标签的全部直接子节点；所以说.contents 属性得到的结果是直接子节点的列表。.children属性返回的是生成器，通过遍历可以得到tag标签中的全部子节点。 (这里一定要记住：不仅是标签算作子节点，字符串也算)
那会找子节点了之后，我们再来看一下怎么找子孙节点:网页内容不变
print(soup.div.descendants)
for i,child in enumerate(soup.div.descendants):
print(i,child)输出结果：
<generator object Tag.descendants at 0x0000018CAAB76900>
0
1 <div>
<h1>The Dormouse's story</h1>
<p class="story">Once upon a time there were three little sisters; and their
names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.
</p>
<p class="story">...</p>
</div>
2
3 <h1>The Dormouse's story</h1>
4 The Dormouse's story
5
6 <p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>,
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
and they lived at the bottom of a well.
</p>
7 Once upon a time there were three little sisters; and their names were
8 <a class="sister" href="http://example.com/elsie" id="link1"><!-- Elsie --></a>
9 Elsie
10 ,
11 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
12 Lacie
13 and
14 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
15 Tillie
16 ;
and they lived at the bottom of a well.
17
18 <p class="story">...</p>
19 ...
20
21通过结果可以看到.descend
最低0.47元/天 解锁文章
扫一扫
709
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
