07_Beautiful Soup一网打尽

激进的猴哥

已于 2024-03-25 09:35:09 修改

阅读量387

点赞数 11

文章标签： python java 开发语言

于 2024-03-25 09:34:40 首次发布

本文链接：https://blog.csdn.net/hcb20010421/article/details/128874721

版权

Beautiful Soup

标签（空格分隔）： Beautiful Soup

简单来说，BeautifulSoup 就是 Python 的一个 HTML 或 XML 的解析库，我们可以用它来方便地从网页

中提取数据，官方的解释如下：

BeautifulSoup 提供一些简单的、Python 式的函数用来处理导航、搜索、修改分析树等功能。它
是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。 BeautifulSoup
自动将输入文档转换为 Unicode 编码，输出文档转换为 utf-8 编码。你不需要考虑
编码方式，除非文档没有指定一个编码方式，这时你仅仅需要说明一下原始编码方式就可以了。 BeautifulSoup 已成为和
lxml、html5lib 一样出色的 Python 解释器，为用户灵活地提供不同的解析策略或强劲的速度。

Beautiful Soup 支持的解析器

安装：


pip install beautifulsoup4

1.案例：

代码：


from bs4 import BeautifulSoup
soup = BeautifulSoup(' Hello ', 'lxml')
print(soup.p.string)

结果：

hello

正文开始

1.节点选择器

直接调用节点按名称，速度非常快

代码：


html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')  #实例化soup对象
print(soup.title)                   # 选择title节点
print(type(soup.title))
print(soup.title.string)    #输出节点的内容
print(soup.head)             # 选择head节点
print(soup.p)           # 选择p节点

结果： html有很多p节点，但是只能匹配第一个


 The Dormouse's story 
<class 'bs4.element.Tag'>
<class 'bs4.element.NavigableString'>
  The Dormouse's story  

The Dormouse's story

2.提取信息：

获取属性：

2.1代码：


html =  html = """
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.attrs)
print(soup.p.attrs['name'])

结果:


{'class': ['title'], 'name': 'dromouse'}
dromouse

2.2获取文本：

代码


html = """
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.string)

结果：


The Dormouse's story

2.3嵌套选择：


html = """
<html><head><title>The Dormouse's story</title></head>
<body>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.head.title) #获取到head 再获取tittle
print(type(soup.head.title))
print(soup.head.title.string)

结果：


 The Dormouse's story 
<class 'bs4.element.Tag'>
The Dormouse's story

2.4关联选择

案例代码：


html = """
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>

2.4.1子节点的获取

代码：


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.contents)

结果： p节包含文本，节点，以列表的形式呈现，但是没有a节点的span标签，所以就是返回子节点，不包含子孙节点

还有另一种方法：chidren，返回生成器类型

代码：


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.children)
for i, child in enumerate(soup.p.children):
  print(i, child)

结果：用for in enumerate 加索引遍历

2.4.2 子孙节点的获取：

代码：


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.descendants)
for i, child in enumerate(soup.p.descendants):
   print(i, child)

结果：

2.4.3 父节点的获取：parent


 html = """
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
        </p>
        <p class="story">...</p>
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.a.parent)

结果：a节点的父节点p被匹配


            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
</p>

2.4.4 所有祖先节点的获取：parents

代码：


 html = """
<html>
    <body>
        <p class="story">
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
        </p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(type(soup.a.parents))
print(list(enumerate(soup.a.parents)))

结果：

2.4.5 获取兄弟节点

代码：


 html = """
<html>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            Hello
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print('Next Sibling', soup.a.next_sibling)  #a的下一个兄弟节点
print('Prev Sibling', soup.a.previous_sibling) #a以前的所有兄弟节点
print('Next Siblings', list(enumerate(soup.a.next_siblings))) 
print('Prev Siblings', list(enumerate(soup.a.previous_siblings)))

结果：

2.4.6 提取数据

代码：


 """
<html>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">Bob</a><a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
        </p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print('Next Sibling:')
print(type(soup.a.next_sibling)) 
print(soup.a.next_sibling)
print(soup.a.next_sibling.string)
print('Parent:')
print(type(soup.a.parents))
print(list(soup.a.parents)[0])
print(list(soup.a.parents)[0].attrs['class'])

结果：

3.方法选择器：

3.1 find_all

查询所有符合的元素：find_all（name ，attrs，recursive，text，**kwargs）


html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(name='ul'))  #查询所有ul节点
print(type(soup.find_all(name='ul')[0]))

结果：


[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
<class 'bs4.element.Tag'>

如果是.tag还能嵌套


for ul in soup.find_all(name='ul'):
print(ul.find_all(name='li'))

结果：



[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

继续嵌套：


for ul in soup.find_all(name='ul'):
       print(ul.find_all(name='li'))
for li in ul.find_all(name='li'):
        print(li.string)

结果：


[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
Foo
Bar
Jay
[<li class="element">Foo</li>, <li class="element">Bar</li>]
Foo
Bar

attrs ：传入属性查询


html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1" name="elements">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
'''

代码：


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(attrs={'id': 'list-1'}))
print(soup.find_all(attrs={'name': 'elements'}))

结果：

text：联合正则匹配节点中文本

代码：


 import re
html='''
<div class="panel">
    <div class="panel-body">
        <a>Hello, this is a link</a>
        <a>Hello, this is a link, too</a>
    </div>
</div>
'''

代码


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(text=re.compile('link')))

结果：


['Hello, this is a link', 'Hello, this is a link, too']

find ：只匹配单个就是符合的第一个

4.css选择器

这个主要用select查询，熟悉css可以用

综合代码：


html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

代码：


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.select('.panel .panel-heading'))
print(soup.select('ul li'))
print(soup.select('#list-2 .element'))
print(type(soup.select('ul')[0]))

结果：

代码：支持嵌套


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
print(ul.select('li'))

结果：


[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

代码：获取属性


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
print(ul['id'])
print(ul.attrs['id'])

结果：


list-1
list-1
list-2
list-2

代码：获取文本


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for li in soup.select('li'):
print('Get Text:', li.get_text())
print('String:', li.string)

结果：


Get Text: Foo
String: Foo
Get Text: Bar
String: Bar
Get Text: Jay
String: Jay
Get Text: Foo
String: Foo
Get Text: Bar
String: Bar

进程已结束，退出代码为 0

激进的猴哥

关注

11
点赞
踩
1

收藏

觉得还不错? 一键收藏
打赏
1
评论
07_Beautiful Soup一网打尽

10分钟看懂
复制链接

扫一扫

07_Beautiful Soup一网打尽

Beautiful Soup

“相关推荐”对你有帮助么？