python3 2018分布式爬虫教程 -6 BeautifulSoup_beautifulsoup分布式爬虫-CSDN博客

本文链接：https://blog.csdn.net/QYmufeng/article/details/85157174

解析库

解析器	使用方法	优势	劣势
Python标准库	BeautifulSoup(markup, "html.parser")	Python的内置标准库、执行速度适中、文档容错能力强	Python 2.7.3 or 3.2.2)前的版本中文容错能力差
lxml HTML 解析器	BeautifulSoup(markup, "lxml")	速度快、文档容错能力强	需要安装C语言库
lxml XML 解析器	BeautifulSoup(markup, "xml")	速度快、唯一支持XML的解析器	需要安装C语言库
html5lib	BeautifulSoup(markup, "html5lib")	最好的容错性、以浏览器的方式解析文档、生成HTML5格式的文档	速度慢、不依赖外部扩展

安装命令：

pip3 install beautifulsoup4

1.基本使用：

# -*- coding:utf-8 -*-

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

# 指定为lxml解析器
soup = BeautifulSoup(html, 'lxml')
# 格式化代码，并将代码补全
html_format = soup.prettify()
print(html_format, "\n----------------------------------")
# 打印title标签的内容
print(soup.title, "\n----------------------------------")
print(soup.title.string)

'''
<html>
 <head>
  <title>
   The Dormouse's story
  </title>
 </head>
 <body>
  <p class="title" name="dromouse">
   <b>
    The Dormouse's story
   </b>
  </p>
  <p class="story">
   Once upon a time there were three little sisters; and their names were
   <a class="sister" href="http://example.com/elsie" id="link1">
    <!-- Elsie -->
   </a>
   ,
   <a class="sister" href="http://example.com/lacie" id="link2">
    Lacie
   </a>
   and
   <a class="sister" href="http://example.com/tillie" id="link3">
    Tillie
   </a>
   ;
and they lived at the bottom of a well.
  </p>
  <p class="story">
   ...
  </p>
 </body>
</html> 
----------------------------------
<title>The Dormouse's story</title> 
----------------------------------
The Dormouse's story
'''

2.选择标签（只返回第一个匹配项,例如一下案例有3个<p></p> 标签，但是只返回第一个）

获取标签内容时，只获取标签内容，自动去除多余的标签（例如：<head><title>The Dormouse's story</title></head> 调用： soup.head.string 返回结果是： The Dormouse's story 而不是： <title>The Dormouse's story</title>）

# -*- coding:utf-8 -*-

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

soup = BeautifulSoup(html, "lxml")
# 格式化并补全 html
html = soup.prettify()
# 获取标签内容
print(soup.head)
print(soup.head.string, "\n-------------------------------")
print(soup.p)
print(soup.p.string)

'''
<head><title>The Dormouse's story</title></head>
The Dormouse's story 
-------------------------------
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
The Dormouse's story
'''

3.获取标签名称（获取标签名称，只返回第一个匹配项）

#coding=utf-8

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

soup = BeautifulSoup(html, "lxml")
# 格式化并补全 html
html = soup.prettify()
# 获取标签名称
print(soup.p.name)

'''
p
'''

4.获取标签属性（获取标签属性，只返回第一个匹配项）

#coding=utf-8

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

soup = BeautifulSoup(html, "lxml")
# 格式化并补全 html
html = soup.prettify()
# 获取标签属性  attrs 可加可不加效果一样
print(soup.p.attrs['name'])
print(soup.p['name'])
print(soup.p.attrs['class'])
print(soup.p['class'])
print("-----------------------------")
print(soup.a['href'])
print(soup.a['class'])

'''
dromouse
dromouse
['title']
['title']
-----------------------------
http://example.com/elsie
['sister']
'''

5.获取标签内容（获取标签内容，只返回第一个标签匹配项）

#coding=utf-8

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

soup = BeautifulSoup(html, "lxml")
# 格式化并补全 html
html = soup.prettify()
# 获取标签内容
print(soup.p.string)

'''
The Dormouse's story
'''

6.标签的嵌套使用

#coding=utf-8

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""

soup = BeautifulSoup(html, "lxml")
# 格式化并补全 html
html = soup.prettify()
# 嵌套使用
print(soup.html.head.title.string)
print(soup.html.head.string)

'''
The Dormouse's story
The Dormouse's story
'''

7.获取子节点内容 contents 获取所有子节点内容，将所有子节点作为一个列表返回（非标签子节点左右的换行符会放在该子节点当作一个子节点返回，两个标签之间的换行符会单独作为一个元素返回）： soup.标签名.contents

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 格式化并补全 html
html = soup.prettify()
# 获取子节点内容
print(soup.p.contents)

'''
['\n            Once upon a time there were three little sisters; and their names were\n            ', <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>, '\n', <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ' \n            and\n            ', <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, '\n            and they lived at the bottom of a well.\n        ']

'''

8.获取子节点及子孙节点（子孙节点：子节点里面的标签）内容作为一个迭代器返回：

soup.标签名.children

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 格式化并补全 html
html = soup.prettify()
# 获取子节点内容
print(soup.p.children)
print(type(soup.p.children))
print("--------------------------------------")
for i, children_node in enumerate(soup.p.children):
    print(i, children_node)

'''
<list_iterator object at 0x000000B59EB96E10>
<class 'list_iterator'>
--------------------------------------
0 
            Once upon a time there were three little sisters; and their names were
            
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4  
            and
            
5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6 
            and they lived at the bottom of a well.

'''

9.获取父节点内容： soup.标签名.parent

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 格式化并补全 html
html = soup.prettify()
# 获取子节点内容
print(soup.a.parent)

'''
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
'''

9.获取祖先点内容： soup.标签名.parents 倒数第二个输出 html标签的内容，倒数第一个输出整个文本内容（一般情况下倒数第二个输出和倒数第一个输出都是一样的）

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 格式化并补全 html
html = soup.prettify()
# 获取子节点内容
print(list(enumerate(soup.a.parents)))

'''
[(0, <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>), 

(1, <body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body>), 

(2, <html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body></html>), 

(3, <html>
<head>
<title>The Dormouse's story</title>
</head>
<body>
<p class="story">
            Once upon a time there were three little sisters; and their names were
            <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> 
            and
            <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
<p class="story">...</p>
</body></html>)]

'''

9.获取兄弟节点：

获取指定标签之前的所有兄弟标签：soup.a.previous_siblings

获取指定标签之后的所有兄弟标签：soup.a.next_siblings

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<html>
    <head>
        <title>The Dormouse's story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 格式化并补全 html
html = soup.prettify()

print(list(enumerate(soup.a.previous_siblings)))
print(list(enumerate(soup.a.next_siblings)))

'''
[(0, '\n            Once upon a time there were three little sisters; and their names were\n            ')]
[(0, '\n'), (1, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>), (2, ' \n            and\n            '), (3, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>), (4, '\n            and they lived at the bottom of a well.\n        ')]

'''

find方法也可以实现以上功能：
soup.li.find_parent()返回li标签的直接父节点

soup.li.find_parents()返回li标签的所有祖先节点

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# soup.li.find_parent()返回li标签的直接父节点
print(soup.li.find_parent())
# soup.li.find_parents()返回所有li标签的祖先节点
print(soup.li.find_parents())

'''
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>, <div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>, <body><div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
</body>, <html><body><div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
</body></html>, <html><body><div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
</body></html>]
'''

soup.li.find_next_siblings()返回li标签后面所有兄弟节点
soup.li.find_next_sibling()返回li标签后面第一个兄弟节点

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 返回li标签后面第一个兄弟节点
print(soup.li.find_next_sibling())
# 返回li标签后面所有兄弟节点
print(soup.li.find_next_siblings())

'''
<li class="element">Bar</li>
[<li class="element">Bar</li>, <li class="element">Jay</li>]
'''

soup.span.find_previous_siblings()返回span标签前面所有兄弟节点

soup.span.find_previous_sibling()返回span标签前面第一个兄弟节点

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">         
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <span>I can endure</span>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 返回span标签前面第一个兄弟节点
print(soup.span.find_previous_sibling())
# 返回span标签前面所有兄弟节点
print(soup.span.find_previous_siblings())

'''
<li class="element">Bar</li>
[<li class="element">Bar</li>, <li class="element">Foo</li>]
'''

find_next()返回该节点后的第一个节点（不论是兄弟节点还是子节点父节点）

find_all_next()返回该节点后的所有节点（不论是兄弟节点还是子节点父节点）

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">         
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <span>I can endure</span>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 返回该节点后的第一个节点（不论是兄弟节点还是子节点父节点）
print(soup.span.find_next())
print("-----------------------------------------")
# 返回该节点后的所有节点（不论是兄弟节点还是子节点父节点）
print(soup.span.find_all_next())

'''
<li class="element">Jay</li>
-----------------------------------------
[<li class="element">Jay</li>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>, <li class="element">Foo</li>, <li class="element">Bar</li>]
'''

find_all_previous()返回节点后所有符合条件的节点

find_previous()返回第一个符合条件的节点

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">         
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <span>I can endure</span>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# find_previous()返回第一个符合条件的节点
print(soup.li.find_previous())
print("-----------------------------------------")
# find_all_previous()返回节点后所有符合条件的节点
print(soup.li.find_all_previous())

'''
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<span>I can endure</span>
<li class="element">Jay</li>
</ul>
-----------------------------------------
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<span>I can endure</span>
<li class="element">Jay</li>
</ul>, <div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<span>I can endure</span>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>, <h4>Hello</h4>, <div class="panel-heading">
<h4>Hello</h4>
</div>, <div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<span>I can endure</span>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>, <body><div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<span>I can endure</span>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
</body>, <html><body><div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<span>I can endure</span>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
</body></html>]
'''

10.标准选择器：find_all( name , attrs , recursive , text , **kwargs )

find_all(name):

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 格式化并补全 html
print(type(soup))

print(soup.find_all(name='ul'))

print(type(soup.find_all(name='ul')))
print(type(soup.find_all(name='ul')[0]))

'''
<class 'bs4.BeautifulSoup'>

[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>, <ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]

<class 'bs4.element.ResultSet'>
<class 'bs4.element.Tag'>
'''

find_all()返回一个bs4.element.ResultSet 的对象，可以对改对象继续使用find_all():

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
# 格式化并补全 html
for ul in soup.find_all('ul'):
    print(ul.find_all('li'))


'''
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
'''

find_all(attrs={}) attrs 传入一个字典类型的参数，指定目标标签的属性，来查找标签

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(attrs={'class':'list list-small','id':'list-2'}))

'''
[<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>]
'''

find_all(id=)：通过指定id的值，来查找目标标签

find_all(class_=)：通过指定class的值，来查找目标标签（注意是 'class_' ，因为class是python的关键词，所以此处class带有下划线）

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(id='list-1'))
print("-----------------------------------")
print(soup.find_all(class_='element'))

'''
[<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>]
-----------------------------------
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
'''

find_all(text=) :通过文本的内容来查找标签本文内容（通常用来匹配文本，不用来查找标签）

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(text='Foo'))

'''
['Foo', 'Foo']
'''

11.find( name , attrs , recursive , text , **kwargs )返回第一个匹配项返回一个bs4.element.Tag的对象：

#coding=utf-8

from bs4 import BeautifulSoup

html = '''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''

# 声明 BeautifulSoup 对象
soup = BeautifulSoup(html, 'lxml')
print(type(soup.find('ul')))
print(soup.find('ul'))
print("----------------------")
print(type(soup.find('li')))
print(soup.find('li'))

'''
<class 'bs4.element.Tag'>
<ul class="list" id="list-1">
<li class="element">Foo</li>
<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
----------------------
<class 'bs4.element.Tag'>
<li class="element">Foo</li>
'''

12.CSS选择器: 通过select()直接传入CSS选择器即可完成选择

. 表示类

# 表示id

#coding=utf-8

from bs4 import BeautifulSoup

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
soup = BeautifulSoup(html, 'lxml')
# 选择标签类名为panel里面的标签类名为panel-heading的所有标签
print(soup.select('.panel .panel-heading'))
# 选择 ul 标签里面所有的 li 标签
print(soup.select('ul li'))
# 选择 id 为list-2标签里面所有类名为element的标签
print(soup.select('#list-2 .element'))
# select 返回一个bs4.element.Tag 对象可以继续使用select
print(type(soup.select('ul')[0]))

'''
[<div class="panel-heading">
<h4>Hello</h4>
</div>]
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>, <li class="element">Foo</li>, <li class="element">Bar</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
<class 'bs4.element.Tag'>
'''

soup.select 返回一个bs4.element.Tag 对象可以继续使用select

soup = BeautifulSoup(html, 'lxml')
# select 返回一个bs4.element.Tag 对象可以继续使用select
print(type(soup.select('ul')[0]))
print("----------------------------------------")
for ul in soup.select('ul'):
    print(ul.select('li'))


'''
<class 'bs4.element.Tag'>
----------------------------------------
[<li class="element">Foo</li>, <li class="element">Bar</li>, <li class="element">Jay</li>]
[<li class="element">Foo</li>, <li class="element">Bar</li>]
'''

13.使用select获取属性格式：ul.["属性名"]

#coding=utf-8

from bs4 import BeautifulSoup

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
    print(ul['id'])
    print(ul.attrs['id'])


'''
list-1
list-1
list-2
list-2
'''

14.使用select获取内容：li.get_text()

#coding=utf-8

from bs4 import BeautifulSoup

html='''
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
'''
soup = BeautifulSoup(html, 'lxml')
for li in soup.select('li'):
    print(li.get_text())


'''
Foo
Bar
Jay
Foo
Bar
'''