爬虫——简单使用Beautiful Soup解析库（一）

皖能

已于 2022-07-31 19:04:09 修改

阅读量1k

点赞数

分类专栏：爬虫 Beautiful Soup 文章标签：爬虫 python

于 2022-07-31 14:19:02 首次发布

本文链接：https://blog.csdn.net/cjz13613855030/article/details/126083229

版权

爬虫同时被 2 个专栏收录

2 篇文章 0 订阅

订阅专栏

Beautiful Soup

2 篇文章 0 订阅

订阅专栏

Beautiful Soup 是一款强大的解析工具，它借助网页的结构和属性等特点来解析网页，可以不用写一些复杂的正则表达式，去提取网页中的某些元素，这在我学习爬虫的过程中，给了我很大的方便，这里就记录一下我的学习笔记，方便我以后复习。（课本内容来自于崔神的网络爬虫开发实战）

一、基本用法

不用多说，直接上代码

html = """
<html><head><title>This is a story</title></head>
<body>
<p class="title" name="dromouse"><b>this is a story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="https://example.com/elsie" class="sister" id="link1"><span>Elsie</span></a>,
<a href="https://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="https://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
# 对BeautifulSoup对象的初始化，并赋值，第二个参数为解析器的类型，这一步自动补全
soup = BeautifulSoup(html,'lxml')
# prettify()方法，把要解析的字符串以标准的缩进格式输出
print(soup.prettify())
# soup.title 选出HTML里的title节点，调用string属性可以得到里面的文本
print(soup.title.string)

这里面简单了解一下，主要是介绍了BeautifulSoup里面的参数含义，第一个参数是要解析的HTML文件，第二个参数是选用的解析器，我使用的是lxml。

然后是prettify()方法，把要解析的字符串以标准的缩进格式输出。

二、节点选择器

在这之前，我先把要解析的HTML文档建立，并命名为dormouse_story.html

<!DOCTYPE html>
<html><head><title>This is a story</title></head>
<body>
<p class="title" name="dromouse"><b>this is a story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a href="https://example.com/elsie" class="sister" id="link1"><span>Elsie</span></a>,
<a href="https://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="https://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>
<p class="story">...</p>

新建一个学习文档，节点选择器.py

(一）选择元素

from bs4 import BeautifulSoup

html = open("dormouse_story.html", "r")
soup = BeautifulSoup(html, "lxml")

"""选择元素"""
print(soup.title)  # 打印输出title节点的选择结果 >> <title>This is a story</title>
print(type(soup.title))  # 输出类型 >> <class 'bs4.element.Tag'>
print(soup.title.string)  # 打印输出title节点内容 >> This is a story
print(soup.head)  # 打印输出head节点及其所有内容  >> <head><title>This is a story</title></head>
print(soup.p)  # 打印输出p节点及其内容，多个p节点只输出第一个 >> <p class="title" name="dromouse"><b>this is a story</b></p>

（二）提取信息

1、获取名称

"""提取信息"""
"""以下选择的p节点都是第一个p节点"""
"获取名称"
# 可以利用name属性获取节点的名称
print(soup.title.name)  # >> title
print(soup.head.name)  # >> head

2、获取属性

"获取属性"
# 每个节点可能有多个属性，调用attrs获取所有属性
print(soup.p.attrs)  # >> {'class': ['title'], 'name': 'dromouse'}
print(soup.p.attrs['name'])  # >> dromouse

# 可以不用attrs，直接获取单个属性
# 一个节点元素可能有多个class属性，输出为列表
# 如果属性值是唯一的，输出结果就是单个字符串
print(soup.p['class'])  # >> ['title']
print(soup.p['name'])  # >> dromouse

3、获取内容

"获取内容"
# 利用string属性获取节点元素包含的内容
print(soup.p.string)  # >> this is a story

（三）嵌套选择

"""嵌套选择"""
# bs4.element.Tag类型可以继续调用节点，进行下一步选择
print(soup.head.title)  # >> <title>This is a story</title>
print(type(soup.head.title))  # >> <class 'bs4.element.Tag'>
print(soup.head.title.string)  # >> This is a story

（四）关联选择

1、子节点和子孙节点

这里，contents属性得到的是节点元素 p 的直接子节点列表，不包含span这个孙节点

html = """<html><head><title>This is a story</title></head>
<body>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a href="https://example.com/elsie" class="sister" id="link1"><span>Elsie</span></a>,
<a href="https://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="https://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>
<p class="story">...</p>"""
"""关联选择"""
"子节点和子孙节点"
soup = BeautifulSoup(html, 'lxml')
# 选取节点元素后，contents属性可以获取它的直接子节点
print(soup.p.contents)
# ['Once upon a time there were three little sisters; and their names were\n    ',
# <a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a>,
# ',\n', 
# <a class="sister" href="https://example.com/lacie" id="link2">Lacie</a>, 
# ' and\n',
# <a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>,
# '\nand they lived at the bottom of a well.']

如果想要得到所有的子孙节点，可以调用descendants属性

print(soup.p.descendants)
for i,child in enumerate(soup.p.descendants):
    print(i, child)

结果如下：

<generator object Tag.descendants at 0x0000028326E04B48>
0 Once upon a time there were three little sisters; and their names were
    
1 <a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a>
2 <span>Elsie</span>
3 Elsie
4 ,

5 <a class="sister" href="https://example.com/lacie" id="link2">Lacie</a>
6 Lacie
7  and

8 <a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>
9 Tillie
10 
and they lived at the bottom of a well.

这次的输出结果就包含了span这个孙节点

2、父节点和祖先节点

parent 父节点

print(soup.a.parent)

<p class="story">Once upon a time there were three little sisters; and their names were
    <a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a>,
<a class="sister" href="https://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>

第一个a的父节点是p节点，这里输出了p节点及其内容

parents 祖先节点

print(soup.a.parents)

<generator object PageElement.parents at 0x0000024EE71E2B48>

可以看到返回结果是生成器类型，用列表输出一下

print(list(enumerate(soup.a.parents)))

[(0, <p class="story">Once upon a time there were three little sisters; and their names were
    <a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a>,
<a class="sister" href="https://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>), 
(1, <body>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a>,
<a class="sister" href="https://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>
<p class="story">...</p></body>), 
(2, <html><head><title>This is a story</title></head>
<body>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a>,
<a class="sister" href="https://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>
<p class="story">...</p></body></html>), 
(3, <!DOCTYPE html>
<html><head><title>This is a story</title></head>
<body>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a>,
<a class="sister" href="https://example.com/lacie" id="link2">Lacie</a> and
<a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>
<p class="story">...</p></body></html>)]

这里，输出的内容就是a节点和祖先节点

3、兄弟节点/同级节点

返回第一个a节点的下一个同级节点，next_sibling

html = """
<html><head><title>This is a story</title></head>
<body>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="https://example.com/elsie" class="sister" id="link1"><span>Elsie</span></a> Hello
<a href="https://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="https://example.com/tillie" class="sister" id="link3">Tillie</a>
and they lived at the bottom of a well.</p>
<p class="story">...</p>"""
soup = BeautifulSoup(html, 'lxml')
# 第一个a节点的下一个同级节点
print('下一个同级节点', soup.a.next_sibling)  # >> 下一个同级节点  Hello

返回第一个a节点的上一个同级节点，previous_sibling

# 第一个a节点的上一个同级节点
print('上一个同级节点', soup.a.previous_sibling)
# >> 上一个节点 Once upon a time there were three little sisters; and their names were

返回第一个a节点的下面所有同级节点，next_siblings

# 返回后面的兄弟节点
print('下面的同级节点', list(enumerate(soup.a.next_siblings)))

下面的同级节点 [(0, ' Hello\n'), 
(1, <a class="sister" href="https://example.com/lacie" id="link2">Lacie</a>), (2, ' and\n'), 
(3, <a class="sister" href="https://example.com/tillie" id="link3">Tillie</a>), 
(4, '\nand they lived at the bottom of a well.')]

返回第一个a节点前面所有同级节点，previous_siblings

# 返回前面的兄弟节点
print('前面的同级节点', list(enumerate(soup.a.previous_siblings)))

前面的同级节点 [(0, 'Once upon a time there were three little sisters; and their names were\n')]

4、提取信息

前面讲解了关于元素节点的选择方法，如果想要获取它们的一些信息，比如文本、属性等，也用同样的方法：

下一个同级节点内容

html = """
<html><head><title>This is a story</title></head>
<body>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="https://example.com/elsie" class="sister" id="link1"><span>Elsie</span></a><a href="https://example.com/lacie" class="sister" id="link2">Lacie</a>
</p>"""
soup = BeautifulSoup(html, 'lxml')
print('下一个同级节点：')
print(soup.a.next_sibling)
print(soup.a.next_sibling.string)

下一个同级节点：
<a class="sister" href="https://example.com/lacie" id="link2">Lacie</a>
Lacie

祖先节点内容

print('祖先节点：')
print(list(soup.a.parents)[0])
print(list(soup.a.parents)[0].attrs['class'])

<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="https://example.com/elsie" id="link1"><span>Elsie</span></a><a class="sister" href="https://example.com/lacie" id="link2">Lacie</a>
</p>
['story']

皖能

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
打赏
0
评论
爬虫——简单使用Beautiful Soup解析库（一）

BeautifulSoup是一款强大的解析工具，它借助网页的结构和属性等特点来解析网页，可以不用写一些复杂的正则表达式，去提取网页中的某些元素，在我学习爬虫的过程中，给了我很大的方便，这里记录一下我的学习笔记，方便以后复习。......
复制链接

扫一扫