BeautifulSoup库详解

最新推荐文章于 2024-07-05 08:36:01 发布
最新推荐文章于 2024-07-05 08:36:01 发布
阅读量140 收藏 1
点赞数
分类专栏： 知识点回顾

安装：pip3 install beautifulsoup4

解析库 

解析器使用方法优势劣势
Python标准库BeautifulSoup(markup,"html.parser")Python的内置标准库、执行速度适中、文档容错能力强Python 2.7.3 or 3.2.2)前的版本中文容错能力差
lxml HTML 解析器BeautifulSoup(markup,"lxml")速度快、文档容错能力强需要安装C语言库
lxml XML 解析器BeautifulSoup(markup,"xml")速度快、唯一支持XML的解析器需要安装C语言库
html5libBeautifulSoup(markup,"html5lib")最好的容错性、以浏览器的方式解析文档、生成HTML5格式的文档速度慢、不依赖外部扩展

基本使用 

html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><！-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.prettify()) #格式化代码的方法
print(soup.title.string)    #打印出title里面的内容

运行结果：
①：
<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters;and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html>
②：The Dormouse's story

标签选择器

选择元素

html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><！-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.title)   #打印出title标签
print(type(soup.title)) #打印出title标签的类型
print(soup.head)    #打印出head标签
print(soup.p)   #打印出p标签

运行结果：
①：<title>The Dormouse's story</title>
②：<class 'bs4.element.Tag'>
③：<head><title>The Dormouse's story</title></head>
④：<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

获取名称

html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><！-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.title.name)  #打印出标签的名称

运行结果：
title

获取属性

html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><！-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.p.attrs['name']) #打印p标签里面的name属性
print(soup.p['name'])   #另一种打印p标签里name属性的方法

运行结果：
dromouse
dromouse

获取内容

html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><！-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.p.string) #打印出p标签里面的内容

运行结果：
The Dormouse's story

嵌套选择

html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters;and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><！-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.head.title.string) #用层层迭代的方式往下选择,选择head里面的title标签的内容

运行结果：
The Dormouse's story

子节点和子孙节点

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.p.contents)  #获取p标签的所有子节点，并以列表的形式返回

运行结果：
['\n            Once upon a time there were three little sisters; and their names were\n            
', <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>, '\n', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, '\n            
and\n            ', <a class="sister" href="http://example.com/tillie" 
id="link3">Tillie</a>, '\n            and they lived at the bottom of a well.\n        ']



PS:下面为另一种方法获取子节点

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.p.children)  #(.children)实际上是一个迭代器，并不是一个列表的形式，需要用循环的方式获取内容，获取子节点
for i,child in enumerate(soup.p.children):  #用enumerate方法将其子节点遍历输出，可以返回节点内容和索引
    print(i,child)

运行结果：
<list_iterator object at 0x0000028DA218BBE0>
0 
            Once upon a time there were three little sisters; and their names were
            
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4 
            and
            
5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6 
            and they lived at the bottom of a well.
ps：获取子孙节点

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.p.descendants)   #返回结果依然是迭代器，获取p标签的所有子节点和子孙节点，会将子孙节点也单独列出来
for i,child in enumerate(soup.p.descendants):
    print(i,child)

运行结果：
<generator object Tag.descendants at 0x0000015CFAA0A318>
0 
            Once upon a time there were three little sisters; and their names were
            
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <span>Elsie</span>
4 Elsie
5 

6 

7 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
8 Lacie
9 
            and
            
10 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
11 Tillie
12 
            and they lived at the bottom of a well.

父节点和祖先节点

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.a.parent)    #获取a标签的父节点

运行结果：
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(list(enumerate(soup.a.parents)))   #(.parents)方法，获取a标签所有的父节点和祖先节点

运行结果：
[(0, <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>), (1, <body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body>), (2, <html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body></html>), (3, <html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body></html>)]

兄弟节点

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a>
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(list(enumerate(soup.a.next_siblings))) #获取后面的兄弟节点
print(list(enumerate(soup.a.previous_siblings)))    #获取前面的兄弟节点

运行结果：
①：[(0, '\n'), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), 
(2, '\n            and\n            '), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, '\n            and they 
lived at the bottom of a well.\n        ')]
②：[(0, '\n            Once upon a time there were three little sisters; and their names 
were\n

标准选择器

find_all(name,attrs,recursive,text,**kwargs)   ※类似于正则表达式，找出所有能匹配成功的结果以列表的形式返回
可根据标签名、属性、内容查找文档

查找name（标签名）：

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.find_all('ul'))  #传入ul标签名，获取所有的ul标签；根据标签名查找
print(type(soup.find_all('ul')[0])) #把ul标签的第一个单独拿出来看一下类型是什么

运行结果：
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
<class 'bs4.element.Tag'>
html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
for ul in soup.find_all('ul'):  #将所有ul标签拿出来遍历
    print(ul.find_all("li"))    #提取出ul标签里面的li标签

运行结果：
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

attrs：根据属性进行查找

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1" name="elements">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.find_all(attrs={'id':'list-1'})) #attrs传入的类型是字典类型，键名是属性名称，键值是属性的值
print(soup.find_all(attrs={'name':'elements'})) #返回的是一个元素的列表

运行结果：
①：[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
②：[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
PS：另一种根据属性查找的方法
html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.find_all(id="list-1")) #可以查找特殊的属性
print(soup.find_all(class_="element"))

运行结果：
①：
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
②：
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li 
class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

text：根据文本的内容进行选择

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.find_all(text="Foo"))    #返回的是标签里面的内容

运行结果：
['Foo', 'Foo']

find(name,attrs,recursive,text,**kwargs) ：使用方法与find_all一样，区别只有返回的元素

find返回单个元素，find_all返回所有元素 

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.find('ul'))
print(type(soup.find('ul')))
print(soup.find('page'))

运行结果：
①：
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
②：
<class 'bs4.element.Tag'>
③：
None

find_parents()：使用方法与find_all一样

返回所有祖先节点 

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumers = soup.find_all(class_ = 'element') #首先找到所有要找父类的标签；如果用find方法，下面这一句可以省略
primaryconsumer = primaryconsumers[0] #将这个标签列表的第一个标签赋值
print(primaryconsumer.find_parents('div')) #查找这个标签的所有祖先节点，以列表的形式返回

运行结果：
[<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>, <div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>]

find_parent()：使用方法与find_all一样

返回直接父亲节点 

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumer = soup.find(class_ = 'element') #定位到属性是element的标签
print(primaryconsumer.find_parent('div')) #查找这个标签的父亲节点

运行结果：
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>

find_next_siblings()：

返回后面所有兄弟节点

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumer = soup.find(class_ = 'element') #定位属性为element的标签
print(primaryconsumer.find_next_siblings('li')) #查找这个标签后面所有的兄弟标签，并以列表的形式输出

运行结果：
[<li class="element">Bar</li>, <li class="element">Jay</li>]

find_next_sibling()：

返回后面第一个兄弟节点

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumer = soup.find(class_ = 'element') #定位属性为element的标签
print(primaryconsumer.find_next_sibling('li')) #查找这个标签后面第一个兄弟标签

运行结果：
<li class="element">Bar</li>

find_previous_siblings()：

返回前面所有的兄弟节点 

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumers = soup.find_all(class_ = 'element') #查找所有属性为element的标签
primaryconsumer = primaryconsumers[2] #定位第三个标签
print(primaryconsumer.find_previous_siblings('li')) #查找这个标签前面所有的兄弟节点

运行结果：
[<li class="element">Bar</li>, <li class="element">Foo</li>]

 find_previous_sibling()：

返回前面第一个兄弟节点 

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumers = soup.find_all(class_ = 'element') #查找所有属性为element的标签
primaryconsumer = primaryconsumers[2] #定位第三个标签
print(primaryconsumer.find_previous_sibling('li')) #查找这个标签前面第一个兄弟节点

运行结果：
<li class="element">Bar</li>

find_all_next()：

返回节点后所有符合条件的节点

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumer = soup.find("div") #定位一个div的标签
print(primaryconsumer.find_all_next('li')) #返回这个标签后面的所有li标签

运行结果：
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li 
class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

 find_next()：

 返回第一个符合条件的节点

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumer = soup.find("div") #定位一个div的标签
print(primaryconsumer.find_next('li')) #返回这个标签后面的第一个li标签

运行结果：
<li class="element">Foo</li>

find_all_previous()：

返回节点前所有符合条件的标签

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumer = soup.find(class_="list") #定位一个属性为list的标签
print(primaryconsumer.find_all_previous('div')) #返回这个标签前面所有的div标签

运行结果：
[<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>, <div class="panel-heading">
<h4>Hello</h4>
</div>, <div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>]

find_previous()：

返回第一个符合条件的标签 

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
primaryconsumer = soup.find(class_="list") #定位一个属性为list的标签
print(primaryconsumer.find_previous('div')) #返回这个标签前面第一个div标签

运行结果：
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>

CSS选择器

通过select()直接传入CSS选择器即可完成选择
html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
print(soup.select('.panel .panel-heading')) #如果传入得是class，选择器前面加.
print(soup.select('ul li')) #如果传入的是标签，前面不需要加入任何内容
print(soup.select("#list-2 .element")) #如果传入得是id，那么前面加#
print(type(soup.select('ul')[0])) #打印输出节点的类型

运行结果：
①：
[<div class="panel-heading">
<h4>Hello</h4>
</div>]
②：
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li 
class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
③：
[<li class="element">Foo</li>, <li class="element">Bar</li>]
④：
<class 'bs4.element.Tag'>  ※：bs4.element.Tag表示可以进行嵌套选择

 

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
for ul in soup.select('ul'):    #层层迭代方法
    print(ul.select('li'))

运行结果：
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li 
class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]

PS：另一种方法

获取属性

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
for ul in soup.select('ul'):
    print(ul['id'])     #获取ul的id属性，结果一下面一致，方法不同结果一致
    print(ul.attrs['id'])

运行结果：
list-1
list-1
list-2
list-2

获取内容

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
from bs4 import BeautifulSoup #引入BeautifulSoup库
soup = BeautifulSoup(html,"lxml") #声明一个BeautifulSoup对象，把html传入，第二个参数传入解析器lxml
for ul in soup.select('li'): #循环遍历li标签
    print(ul.get_text())    #用get_text方法获取标签里面的文本

运行结果：
Foo
Bar
Jay
Foo
Bar

总结：

  •  推荐使用lxml解析库，必要时使用html.parser
  • 标签选择筛选功能弱但是速度快
  • 建议使用find()、find_all()查询匹配单个结果或多个结果
  • 如果对CSS选择器熟悉建议使用select()
  • 记住常用的获取属性和文本值得方法
tjh0001
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
python中BeautifulSoup库详解，网页分析，安装、使用、分享学习
weixin_57712785的博客
10-20 205
# BeautifulSoup # 灵活方便的网页解析 # pip3 install BeautifulSoup4 # 用法讲解 # lxml html 解析 # lxml XML 解析 # html5lib 解析 # BeautifulSoup from bs4 import BeautifulSoup soup = BeautifulSoup(html,'lxml') print(soup.prettify()) #自动补全HTML代码 print(soup.title.string) #获取里面.
python beautiful soup库的用法
weixin_34127717的博客
03-09 3026
参考：http://cuiqingcai.com/1319.html Beautiful Soup 4.2.0 文档 1. Beautiful Soup 简介 简单来说，Beautiful Soup是python的一个库，最主要的功能是从网页抓取数据。官方解释如下： Beautiful Soup提供一些简单的、python式的函数用来处理导航、搜索...
网络爬虫之BeautifulSoup参数详解
最新发布
载_酒i
07-05 1499
本文详细介绍了 BeautifulSoup 的初始化方法、解析器选择、常用参数和属性以及常用方法。通过这些知识，你可以更加高效地进行网页解析和数据提取。希望本文对你有所帮助，如果你有任何问题或建议，欢迎留言讨论。BeautifulSoup 官方文档通过阅读本文，你应该能够更加自信地使用 BeautifulSoup 进行网页数据的抓取和解析。
五 . BeautifulSoup库详解及运用
niandanzhi9501的博客
10-08 522
BeautifulSoup库详解及运用BeautifulSoup是灵活又方便的网页解析库，处理高效，支持多种解析器，利用它不用编写正则表达式即可实现网点信息的提取。解析库各个解析库的比较： 解析器 使用方法 优势 劣势 Python标准库 BeautifulSoup(markup, “html.parser”) Python的内置标准库、执行速度适中 、文档容错能力强
beautifulsoup详解
qh075的博客
09-30 2593
1，用CSS选择器时，标签名不加任何修饰，class类名前加. , id名前加#2，用到的方法是soup.select()，返回类型是list。3，多个过滤条件需要用空格隔开,严格遵守从前往后逐层筛选。
beautifulSoup详解
weixin_45203459的博客
10-17 2012
BeautifulSoup Beautiful Soup和lxml、html6lib同样作为python解释器，为用户灵活地提供不同的解析策略或强劲的速度。BeautifulSoup是一个工具箱,通过解析文档为用户提供需要抓取的数据,操作简单，代码简洁 能自动补全代码中确实的标签 第一种用法 创建 beautifulsoup 对象： soup=BeautifulSoup(html,‘lxml’) ...
课时11：BeautifulSoup库详解.rar
03-01
Python3 爬虫实战Python3 爬虫实战Python3 爬虫实战Python3 爬虫实战Python3 爬虫实战Python3 爬虫实战Python3 爬虫实战Python3 爬虫实战Python3 爬虫实战
BeautifulSoup用法详解1
08-08
BeautifulSoup 是一个 Python 的库，专门用于解析 HTML 和 XML 文档，从而方便地提取和操作数据。它是一个强大的工具，尤其适合网页抓取和数据挖掘任务。BeautifulSoup 提供了一系列简洁的 Python 函数，使得解析...
beautifulsoup用法详解.zip
05-23
以下是对BeautifulSoup库的深入讲解。 1. **安装BeautifulSoup** 在Python环境中，可以通过pip命令来安装BeautifulSoup。在命令行输入`pip install beautifulsoup4` 和 `pip install requests`（因为BeautifulSoup...
BeautifulSoup用法详解.pdf
05-23
### BeautifulSoup 用法详解 #### 一、引言 BeautifulSoup 是一款强大的 Python 库，专为处理 HTML 和 XML 文件而设计。它不仅能够解析这些文件，还特别擅长处理那些结构不规范或存在错误的文档，这对于实际网页...
python3实现网络爬虫之BeautifulSoup使用详解
09-19
首先，安装BeautifulSoup库是非常简单的。由于它不是Python的标准库，我们需要通过第三方工具如pip或集成开发环境（IDE）如PyCharm来安装。在PyCharm中，你可以通过设置 -> 项目 -> 项目解释器界面找到添加新包的...
BeautifulSoup详解
热门推荐
trayvontang的博客
10-30 1万+
目录一、BeautifulSoup构建1.1 通过字符串构建1.2 从文件加载二、Tag对象2.1 string、strings、stripped_strings2.2 get_text()2.3 属性三、contents、children与descendants3.1 parent、parents3.2 next_sibling、previous_sibling3.3 next_element、...
【Python爬虫】详解BeautifulSoup()及其方法
小吉妙妙屋
03-19 2896
使用 BeautifulSoup，你可以将 HTML 或 XML 文档加载到解析树中，并使用类似于 DOM（文档对象模型）的方式来遍历和搜索文档的结构。然后，你可以使用各种方法和属性来定位、提取和操作文档中的元素和数据。总的来说，BeautifulSoup 是一个功能强大且易于使用的工具，用于解析和处理 HTML、XML 等文档，并从中提取所需的数据。由于link标签里面的链接中有a字母，所以link标签也被选出来了。我们查找a标签，head标签里面有a这个 字母，所以被选出来了。
爬虫笔记：BeautifulSoup详解
总裁余（余登武）博客
01-27 8469
BeautifulSoup讲解。标签选择筛选功能弱但是速度快 建议使用find()、find_all() 查询匹配单个结果或者多个结果 如果对CSS选择器熟悉建议使用select() 记住常用的获取属性和文本值的方法。语言python
爬虫（九十）Beautiful Soup 详解(一)
人生代码 ---- 公众号
03-29 571
春季，四季之一。春，代表着温暖、生长。春季，阴阳之气开始转变，万物随阳气上升而萌牙生长，大地呈现春和景明之象创建 Beautiful Soup 对象首先...
python3 BeautifulSoup库介绍
星落的博客
06-28 1529
BeautifulSoup是python的一个库，最主要的功能就是从网页爬取我们 需要的数据。 BeautifulSoup将html解析为对象进行处理，全部页面转换为字典或数组，
自学python爬虫（五）BeautifulSoup库的介绍
浮生半醉的博客
08-06 534
一、概念 BeautifulSoup是灵活又方便的网页解析库，处理高效。支持多种解释器。利用它可以不用编写正则表达式即可方便地实现网页信息的提取。 二、解析库 三、实例讲解 下面用到的代码 html = """&lt;html&gt; &lt;head&gt; &lt;title&gt;关于软件学院本科生开放实验室注册的通知-中南大学软件学院&lt;/title&gt;&lt...
Python网络爬虫：BeautifulSoup库详解
"第二讲整理1 - Python网络爬虫中的BeautifulSoup库入门" 在Python的网络爬虫领域，BeautifulSoup库是一个非常重要的工具，它主要用于解析HTML和XML文档，使得我们能够方便地提取和操作网页数据。本讲主要介绍了...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值