Beautiful Soup库

最新推荐文章于 2024-04-17 13:52:02 发布

Z_sam

最新推荐文章于 2024-04-17 13:52:02 发布

阅读量193

点赞数

分类专栏： python库

本文链接：https://blog.csdn.net/u010553139/article/details/102570515

版权

python库专栏收录该内容

11 篇文章 0 订阅

订阅专栏

Beauti ful Soup 就是Python 的一个HTML 或XML 的解析库，可以用它来方便地从网页中提取数据
https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/

初始化

from bs4 import BeautifulSoup
soup = BeautifulSoup(html,‘lxml’)
或soup = BeautifulSoup(open(’./bsTest01.html’),‘lxml’)

解析器

1节点选择器

from bs4 import BeautifulSoup

##1节点选择器
html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="linkl"><!-- Elsie --></a>,
<a href="http://example.com/elsie" class="sister" id="link2">Lacie</a> and 
<a href="http://example.com/elsie" class="sister" id="link3">Tillie</a>;
and they live at the bottom of a well.</p>
<p class="story">...</p>
'''
soup = BeautifulSoup(html,'lxml')
#选择元素
print(soup.title)
print(type(soup.title))
print(soup.title.string)
print(soup.head)
print(soup.p)
--------------
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story
<head><title>The Dormouse's story</title></head>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>

bs4.element.Tag类型属性

提取信息

获取节点名称

print(soup.title.name)
--------------------------
title

获取节点属性.attrs可以获取所有属性

print(soup.p.attrs)
print(soup.p.attrs['name'])
#另外一种写法：由于attrs返回结果是字典，可以从字典中获取某个键值。
print(soup.p['name'])
print(soup.p['class'])
--------------------------
{'class': ['title'], 'name': 'dromouse'}
dromouse
dromouse
['title']

注意判断结果值，有些返回是字典有些是字符串

获取内容

print(soup.p.string)
--------------------------
The Dormouse's story

嵌套选择

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
"""
soup = BeautifulSoup(html,'lxml')
print(soup.head.title)
print(type(soup.head.title))
print(soup.head.title.string)
------------------------------------
<title>The Dormouse's story</title>
<class 'bs4.element.Tag'>
The Dormouse's story

嵌套每次返回结果都是’bs4.element.Tag’类型，然后可以继续调用节点进行下一步的选择。如：获取了head节点元素，继续调用head来选取其内部的节点元素

关联选择

html = '''
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="linkl"><span>Elsie<span></a>,
<a href="http://example.com/elsie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/elsie" class="sister" id="link3">Tillie</a>;
and they live at the bottom of a well.</p>
<p class="story">...</p>
'''
soup = BeautifulSoup(html,'lxml')

子节点和子孙节点。注：contents获取子节点

print(soup.p.contents)
---------------------------------
['Once upon a time there were three little sisters; and their names were\n', <a class="sister" href="http://example.com/elsie" id="linkl"><span>Elsie<span></span></span></a>, ',\n', <a class="sister" href="http://example.com/elsie" id="link2">Lacie</a>, ' and\n', <a class="sister" href="http://example.com/elsie" id="link3">Tillie</a>, ';\nand they live at the bottom of a well.']

children子节点，但结果是生成器类型，用for输出

print(soup.p.children,type(soup.p.children))
for i,child in enumerate(soup.p.children): #enumerate()函数用于将一个可遍历的数据对象(如列表、元组或字符串)组合为一个索引序列，同时列出数据和数据下标，一般用在 for 循环当中
    print(i,child)
--------------------
<list_iterator object at 0x0000000002FD0128> <class 'list_iterator'>
0 Once upon a time there were three little sisters; and their names were

1 <a class="sister" href="http://example.com/elsie" id="linkl"><span>Elsie<span></span></span></a>
2 ,

3 <a class="sister" href="http://example.com/elsie" id="link2">Lacie</a>
4  and

5 <a class="sister" href="http://example.com/elsie" id="link3">Tillie</a>
6 ;
and they live at the bottom of a well.

descendants子孙节点，生成器类型，用for输出

print(soup.p.descendants,type(soup.p.descendants))
for i,child in enumerate(soup.p.descendants):
    print(i,child)
------------------------
<generator object descendants at 0x000000000067F570> <class 'generator'>
0 Once upon a time there were three little sisters; and their names were

1 <a class="sister" href="http://example.com/elsie" id="linkl"><span>Elsie<span></span></span></a>
2 <span>Elsie<span></span></span>
3 Elsie
4 <span></span>
5 ,

6 <a class="sister" href="http://example.com/elsie" id="link2">Lacie</a>
7 Lacie
8  and

9 <a class="sister" href="http://example.com/elsie" id="link3">Tillie</a>
10 Tillie
11 ;
and they live at the bottom of a well.

父节点和祖先节点

父节点parent

print(soup.a.parent,type(soup.a.parent))
----------------
<p class="story">Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="linkl"><span>Elsie<span></span></span></a>,
<a class="sister" href="http://example.com/elsie" id="link2">Lacie</a> and
<a class="sister" href="http://example.com/elsie" id="link3">Tillie</a>;
and they live at the bottom of a well.</p> <class 'bs4.element.Tag'>

祖先节点

print(soup.a.parents,type(soup.a.parents))
print(list(enumerate(soup.a.parents)),'\n')#转成列表
for i,parent in enumerate(soup.a.parents): #把结果历遍出来
    print(i,parent)
---------------------
<generator object parents at 0x000000000090F4C0> <class 'generator'>
结果略

注意：结果为生成器类型

兄弟节点

html = '''
<html>
<body>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="linkl"><span>Elsie<span></a> Hello
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they live at the bottom of a well.
</p>
'''
soup = BeautifulSoup(html,'lxml')

下一个兄弟节点

print('Next Sibling:',soup.a.next_sibling)
-------------------------------------------------
Next Sibling:  Hello

上一个兄弟节点

print('prev Sibling:',soup.a.previous_sibling)
----------------------------------------------------
prev Sibling: Once upon a time there were three little sisters; and their names were

下一个所有兄弟节点

print('Next Sibling:',list(enumerate(soup.a.next_siblings)))
-----------------------------------
Next Sibling: [(0, ' Hello\n'), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (2, ' and\n'), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, ';\nand they live at the bottom of a well.\n')]

上一个所有兄弟节点

print('prev Sibling:',list(enumerate(soup.a.previous_siblings)))
--------------------------------
prev Sibling: [(0, 'Once upon a time there were three little sisters; and their names were\n')]

提取信息

html = '''
<html>
<body>
<p class="story">
    Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="linkl">Bob</a><a href="http://example.com/lacie" 
class="sister" id="link2">Lacie</a> 
</p>
'''
soup = BeautifulSoup(html,'lxml')
print(type(soup.a.next_sibling),soup.a.next_sibling)
print(soup.a.next_sibling.string)
print(type(soup.a.parents))
print(list(soup.a.parents)[0])
print(list(soup.a.parents)[0].attrs['class'])
-------------------------------------------
<class 'bs4.element.Tag'> <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
Lacie
<class 'generator'>
<p class="story">
    Once upon a time there were three little sisters; and their names were
<a class="sister" href="http://example.com/elsie" id="linkl">Bob</a><a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
</p>
['story']

方法选择器

find_all()

name 根据节点名称进行查找

from bs4 import BeautifulSoup

html = """
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div>
        <div class="panel-body">
            <ul class="list" id="list-1">
                <li class="element">Foo</li>
                <li class="element">Bar</li>
                <li class="element">Jay</li>
            </ul>
            <ul class="list list-small" id="list-2">
                <li class="element">Foo</li>
                <li class="element">Bar</li>
            </ul>
        </div>
    </div>
</div>
"""
soup = BeautifulSoup(html,'lxml')
#find_all()
print(soup.find_all(name='ul'))
print(type(soup.find_all(name='ul')[0]))
-------------
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
<class 'bs4.element.Tag'>

for ul in soup.find_all(name='ul'):
    print(ul.find_all(name='li'))
    for li in ul.find_all(name='li'):
        # print(type(li))
        print(li.string)
---------------------------
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
Foo
Bar
Jay
[<li class="element">Foo</li>, <li class="element">Bar</li>]
Foo
Bar

sttrs 根据属性来查询

html = """
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div>
        <div class="panel-body">
            <ul class="list" id="list-1" name="elements">
                <li class="element">Foo</li>
                <li class="element">Bar</li>
                <li class="element">Jay</li>
            </ul>
            <ul class="list list-small" id="list-2">
                <li class="element">Foo</li>
                <li class="element">Bar</li>
            </ul>
        </div>
    </div>
</div>
"""
soup = BeautifulSoup(html,'lxml')
###attrs根据属性来查询
print(soup.find_all(attrs={"id":"list-1"}))
print(soup.find_all(attrs={'name':'elements'}))
#id 、class 可以直接查
print(soup.find_all(id='list-1'))
print("---"*10)
print(soup.find_all(class_='element'))
------------------------
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
[<ul class="list" id="list-1" name="elements">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
------------------------------
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]

find_parents()返回所有祖先节点.
find_parent()返回父节点
find_next_siblings()返回后面所有兄弟节点。
find_next_sibling()返回后面兄弟节
find_previous_siblings()返回前面所有兄弟节点
find_previous_sibling()返回前面兄弟节点
find_all_next()返回节点后所有符合条件的节点
find_next()返回第一个符合条件节点
find_all_previous()返回节点前所有符合条件的节点
find_previous()返回节点前第一个符合条件的节点

CSS选择器

html = """
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div>
        <div class="panel-body">
            <ul class="list" id="list-1">
                <li class="element">Foo</li>
                <li class="element">Bar</li>
                <li class="element">Jay</li>
            </ul>
            <ul class="list list-small" id="list-2">
                <li class="element">Foo</li>
                <li class="element">Bar</li>
            </ul>
        </div>
    </div>
</div>
"""
soup = BeautifulSoup(html,'lxml')
print(soup.select('.panel .panel-heading'))
print(soup.select('ul li'))
print(soup.select('#list-2 .element'))
print(type(soup.select('ul')[0]))
-------------------------
[<div class="panel-heading">
<h4>Hello</h4>
</div>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
<class 'bs4.element.Tag'>

嵌套选择

for ul in soup.select('ul'):
    print(ul)
--------------------
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>

获取属性

for ul in soup.select('ul'):
    print(ul['id'])
-----------
list-1
list-2

获取文本

for li in soup.select('li'):
    print('get text:',li.get_text())
    print('string：',li.string)
----------
get text: Foo
string： Foo
get text: Bar
string： Bar
get text: Jay
string： Jay
get text: Foo
string： Foo
get text: Bar
string： Bar

Z_sam

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Beautiful Soup库

Beauti ful Soup 就是Python 的一个HTML 或XML 的解析库，可以用它来方便地从网页中提取数据https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/初始化from bs4 import BeautifulSoupsoup = BeautifulSoup(html,‘lxml’)或soup = BeautifulSoup...
复制链接

扫一扫