五 . BeautifulSoup库详解及运用

最新推荐文章于 2024-08-26 00:35:17 发布
最新推荐文章于 2024-08-26 00:35:17 发布
阅读量548 收藏 1
点赞数 1
分类专栏： Python从入门到放弃系列 文章标签： python 库 爬虫
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/niandanzhi9501/article/details/78173309
版权

BeautifulSoup库详解及运用

BeautifulSoup是灵活又方便的网页解析库，处理高效，支持多种解析器，利用它不用编写正则表达式即可实现网点信息的提取。

解析库

各个解析库的比较：

解析器使用方法优势劣势
Python标准库BeautifulSoup(markup, “html.parser”)Python的内置标准库、执行速度适中 、文档容错能力强Python 2.7.3 or 3.2.2)前的版本中文容错能力差
lxml HTML 解析器BeautifulSoup(markup, “lxml”)速度快、文档容错能力强需要安装C语言库
lxml XML 解析器BeautifulSoup(markup, “xml”)速度快、唯一支持XML的解析器需要安装C语言库
html5libBeautifulSoup(markup, “html5lib”)最好的容错性、以浏览器的方式解析文档、生成HTML5格式的文档速度慢、不依赖外部扩展

基本使用

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.prettify())
print(soup.title.string)

结果为：

<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!-- Elsie -->
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>
The Dormouse's story

自动把代码转化成标准的lxml格式的文件。

标签选择器

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.title)
print(type(soup.title))
print(soup.head)
print(soup.p)

结果为：

<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
<head><title>The Dormouse's story</title></head>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

获取属性

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.attrs['name'])
print(soup.p['name'])

结果：dromouse
dromouse

获取内容

用.string方法获取内容。

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p clss="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.string)

嵌套选择

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.head.title.string)

子节点和子孙节点

content方法可以提取该标签下所有的子节点。

html = """
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.contents)

输出结果为：

[u'\n            Once upon a time there were three little sisters; and their names were\n            ', <a class="sister" href="http://example.com/elsie" id="link1">\n<span>Elsie</span>\n</a>, u'\n', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, u' \n            and\n            ', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, u'\n            and they lived at the bottom of a well.\n        ']
In [8]:

以上结果返回为list类型，有点杂乱，用迭代的方法依次取出。

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.children)
for i, child in enumerate(soup.p.children):
    print(i, child)

结果为：


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.children)
for i, child in enumerate(soup.p.children):
    print(i, child)
html = """
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.children)
for i, child in enumerate(soup.p.children):
    print(i, child)
<list_iterator object at 0x1064f7dd8>
0 
            Once upon a time there were three little sisters; and their names were

1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4  
            and

5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6 
            and they lived at the bottom of a well.

descendants

获取所有的子孙节点

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.p.descendants)
for i, child in enumerate(soup.p.descendants):
    print(i, child)

parent

用来获得父节点。

html = """
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.a.parent)

结果会把a标签的父标签-p标签,输出出来。

parents方法

获得祖先节点，和获得子孙节点正好相反。

兄弟节点

用.next_siblings或许上一个兄弟节点。
用.previous_siblings获取上一个兄弟节点。

标准选择器

find_all( name , attrs , recursive , text , **kwargs )

可根据标签名、属性、内容查找文档

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all('ul'))
print(type(soup.find_all('ul')[0]))

findall()方法查找所有指定的标签。
结果就是ul标签

[<ul class="list" id="list-1">\n<li class="element">Foo</li>\n<li class="element">Bar</li>\n<li class="element">Jay</li>\n</ul>,
<ul class="list list-small" id="list-2">\n<li class="element">Foo</li>\n<li class="element">Bar</li>\n</ul>]
<class 'bs4.element.Tag'>

找到所有ul标签下的li标签的代码：

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.find_all('ul'):
    print(ul.find_all('li'))

结果为：

[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

attrs

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1" name="elements">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(attrs={'id': 'list-1'}))
print(soup.find_all(attrs={'name': 'elements'}))

attrs利用字典类型，结果如下：

[<ul class="list" id="list-1" name="elements">\n<li class="element">Foo</li>\n<li class="element">Bar</li>\n<li class="element">Jay</li>\n</ul>]
[<ul class="list" id="list-1" name="elements">\n<li class="element">Foo</li>\n<li class="element">Bar</li>\n<li class="element">Jay</li>\n</ul>]

支持直接用id和element查找

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(id='list-1'))
print(soup.find_all(class_='element')) //class是关键字所以要用class_=" "

结果为：

[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

利用text进行选择

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(text='Foo'))

查找文本

['Foo', 'Foo']

找的是text内容而不是标签。

find

find用法和findall一模一样，但是返回的是找到的第一个符合条件的内容输出

CSS选择器

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.select('.panel .panel-heading'))
print(soup.select('ul li')) //选择ul标签下面的li标签
print(soup.select('#list-2 .element')) //通过#number选择ID
//查找class=element的id=list-2的标签
print(type(soup.select('ul')[0]))

[<div class="panel-heading">
<h4>Hello</h4>
</div>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
<class 'bs4.element.Tag'>

获取属性

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
    print(ul['id'])
    print(ul.attrs['id'])

结果：
用[ ]即可获取属性。ul[id]获取ul的id属性

list-1
list-1
list-2
list-2

获取内容

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for li in soup.select('li'):
    print(li.get_text())

只要用get_text函数就能获取内容了。

Foo
Bar
Jay
Foo
Bar

总结

  • 推荐使用lxml解析库，必要时使用html.parser
  • 标签选择筛选功能弱但是速度快
  • 建议使用find()、find_all() 查询匹配单个结果或者多个结果
  • 如果对CSS选择器熟悉建议使用select()
  • 记住常用的获取属性和文本值的方法
GenjieLi
关注
专栏目录
[Python从零到壹] 五.网络爬虫之BeautifulSoup基础语法万字详解
杨秀璋的专栏
11-08 1万+
欢迎大家来到“Python从零到壹”，在这里我将分享约200篇Python系列文章，带大家一起去学习和玩耍，看看Python这个有趣的世界。所有文章都将结合案例、代码和作者的经验讲解，真心想把自己近十年的编程经验分享给大家，希望对您有所帮助，文章中不足之处也请海涵。Python系列整体框架包括基础语法10篇、网络爬虫30篇、可视化分析10篇、机器学习20篇、大数据分析20篇、图像识别30篇、人工智能40篇、Python安全20篇、其他技巧10篇。您的关注、点赞和转发就是对秀璋最大的支持，知识无价人有情，希望
beautifulSoup详解
weixin_45203459的博客
10-17 2063
BeautifulSoup Beautiful Soup和lxml、html6lib同样作为python解释器，为用户灵活地提供不同的解析策略或强劲的速度。BeautifulSoup是一个工具箱,通过解析文档为用户提供需要抓取的数据,操作简单，代码简洁 能自动补全代码中确实的标签 第一种用法 创建 beautifulsoup 对象： soup=BeautifulSoup(html,‘lxml’) ...
python中BeautifulSoup库详解，网页分析，安装、使用、分享学习
weixin_57712785的博客
10-20 246
# BeautifulSoup # 灵活方便的网页解析 # pip3 install BeautifulSoup4 # 用法讲解 # lxml html 解析 # lxml XML 解析 # html5lib 解析 # BeautifulSoup from bs4 import BeautifulSoup soup = BeautifulSoup(html,'lxml') print(soup.prettify()) #自动补全HTML代码 print(soup.title.string) #获取里面.
BeautifulSoup库详解
qq_39748755的博客
09-15 157
安装：pip3 install beautifulsoup4 解析库  解析器 使用方法 优势 劣势 Python标准库 BeautifulSoup(markup,"html.parser") Python的内置标准库、执行速度适中、文档容错能力强 Python 2.7.3 or 3.2.2)前的版本中文容错能力差 lxml HTML 解析器 ...
BeautifulSoup对象select()方法之gpt介绍
最新发布
AQUILIOS的博客
08-26 553
select()方法非常灵活，可以用来根据标签、类、ID、属性以及它们的组合来选择元素。你可以使用 CSS 选择器的几乎所有特性，从而高效地提取你需要的信息。
python beautiful soup库的用法
weixin_34127717的博客
03-09 3173
参考：http://cuiqingcai.com/1319.html Beautiful Soup 4.2.0 文档 1. Beautiful Soup 简介 简单来说，Beautiful Soup是python的一个库，最主要的功能是从网页抓取数据。官方解释如下： Beautiful Soup提供一些简单的、python式的函数用来处理导航、搜索...
BeautifulSoup库的应用
hkds0512的博客
07-19 552
beautifulSoup库，是一个非常流行的python模块。通过BeautifulSoup库可以轻松地解析Requests库请求的网页，并把网页源代码解析为Soup文档，以便过滤提取数据。 import requests from bs4 import BeautifulSoup headers = {‘User-Agent’:‘Mozilla/5.0 (Windows NT 6.1; Win...
beautifulsoup用法详解.zip
05-23
以下是对BeautifulSoup库的深入讲解。 1. **安装BeautifulSoup** 在Python环境中，可以通过pip命令来安装BeautifulSoup。在命令行输入`pip install beautifulsoup4` 和 `pip install requests`（因为BeautifulSoup...
BeautifulSoup用法详解1
08-08
在实际应用中，结合 requests 库抓取网页内容，再用 BeautifulSoup 处理，可以构建出强大的网络数据处理工具。 总之，BeautifulSoup 是 Python 中解析 HTML 和 XML 的利器，它的简洁 API 和强大的功能使其成为网页...
BeautifulSoup用法详解.pdf
05-23
### BeautifulSoup 用法详解 #### 一、引言 BeautifulSoup 是一款强大的 Python 库，专为处理 HTML 和 XML 文件而设计。它不仅能够解析这些文件，还特别擅长处理那些结构不规范或存在错误的文档，这对于实际网页...
北理Python爬虫：BeautifulSoup库详解及实战应用
本资源是关于北京大学理学院开设的Python爬虫课程的一部分，专注于BeautifulSoup库的学习。BeautifulSoup是一个在Python中广泛使用的库，用于解析HTML和XML文档，帮助开发者从复杂网页结构中提取所需的数据。它使得...
BeautifulSoup库的使用
专注于Python编程技术的分享与交流，致力于帮助开发者提升编程技能，解决实际问题，探索Python的无限可能。
08-13 2353
BeautifulSoup库是python中常用的的网页解析库，灵活，高效，支持多种解析器。可以很方便地实现网页信息的提取。
beautifulsoup详解
qh075的博客
09-30 2653
1，用CSS选择器时，标签名不加任何修饰，class类名前加. , id名前加#2，用到的方法是soup.select()，返回类型是list。3，多个过滤条件需要用空格隔开,严格遵守从前往后逐层筛选。
BeautifulSoup详解
大数据 小白学习录
04-20 2459
文章目录1.下载安装2.导入3.装载HTML文档4.将文档数转换成字符串格式5.BeautifulSoup查找文档元素6.BeautifulSoup遍历文档树7.BeautifulSoup使用css语法查找元素9.字符编码问题8.实例：爬取中国天气网数据兰州7天的 1.下载安装 pip install bs4 2.导入 from bs4 import BeautifulSoup as bs 3.装载HTML文档 soup = bs(doc, 'lxml') #doc是一个HTML文档字符串，可以自动补.
爬虫笔记：BeautifulSoup详解
总裁余（余登武）博客
01-27 8525
BeautifulSoup讲解。标签选择筛选功能弱但是速度快 建议使用find()、find_all() 查询匹配单个结果或者多个结果 如果对CSS选择器熟悉建议使用select() 记住常用的获取属性和文本值的方法。语言python
爬虫（九十）Beautiful Soup 详解(一)
人生代码 ---- 公众号
03-29 582
春季，四季之一。春，代表着温暖、生长。春季，阴阳之气开始转变，万物随阳气上升而萌牙生长，大地呈现春和景明之象创建 Beautiful Soup 对象首先...
BeautifulSoup库
萨丁
09-02 532
python3爬虫学习笔记 BeautifulSoup 基本使用 html = ''' <html><head><title>The Dormouse's story</title></head> <body> <p class="title" name="dromouse"><b>The Dormou...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值