Python Beautiful Soup库简单讲解--包括安装

最新推荐文章于 2024-08-08 15:42:55 发布

Camilleferros

最新推荐文章于 2024-08-08 15:42:55 发布

阅读量491

点赞数

分类专栏： Python

本文链接：https://blog.csdn.net/Huangpengyu123/article/details/106960613

版权

Python 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

Beautiful Soup库入门

本文是Mooc上 Python网络爬虫与信息提取的笔记

1 Beautiful Soup库的安装

这里以PyCharm为例(因为我用的就是这种方法)

点击的顺序为

File->Settings->Project:Pycharm->Python Interperter->

然后点击右上角的加号.添加我们想要加入的库

2 Beautiful Soup库的基本元素

一个HTML文件是由一组<>构成的标签组成的

Beautiful Soup库就是用来处理这些标签的

<p>..</p> : 标签  Tag

<p class=“title”> … </p>

名称  Name   属性  Attributes
成对出现			0个或者多个

Beautiful Soup库的引用

Beautiful Soup库，也叫 beautifulsoup4或者bs4

引用方式如下

 from bs4 import BeautifulSoup

HTML文档标签树 BeautifulSoup 是相等的

BeautifulSoup对应一个HTML/XML文档的全部内容

BeautifulSoup库的解析器

soup = BeautifulSoup(‘data’，‘html.parser’)

‘html.parser’ 就是更改的内容

解析器	使用方法	条件
bs4的HTML解析器	BeautifulSoup(mk,‘html.parser’)	安装bs4库
lxml的HTML解析器	BeautifulSoup(mk,‘lxml’)	pip install lxml
lxml的XML解析器	BeautifulSoup(mk,‘xml’)	pip install lxml
html5lib的解析器	BeautifulSoup(mk,‘html5lib’)	pip install html5lib

BeautifulSoup类的基本元素

<p class=“title”> … </p>

基本元素	说明
Tag	标签，最基本的信息组织单元，分别用<>和</>标明开头和结尾
Name	标签的名字， … 的名字是’p’，格式：.name
Attributes	标签的属性，字典形式组织，格式：.attrs
NavigableString	标签内非属性字符串，<>…</>中字符串，格式：.string
Comment	标签内字符串的注释部分，一种特殊的Comment类型

一堆例子

Tag的使用

import  requests
from bs4 import BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text   #获取到url 地点的源码
soup = BeautifulSoup(demo,'html.parser')
print(soup.title)	#输出title标签的内容

结果如下

<title>This is a python demo page</title>

查看标签的名字

import  requests
from bs4 import BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text   #获取到url 地点的源码
soup = BeautifulSoup(demo,'html.parser')
print(soup.a.name)                      #输出a标签的名字
print(soup.a.parent.name)               #输出a标签的父标签的名字
print(soup.a.parent.parent.name)        #输出a标签的父标签的父标签的名字

结果如下

a
p
body

标签的属性信息

import  requests
from bs4 import BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text   #获取到url 地点的源码
soup = BeautifulSoup(demo,'html.parser')
tag = soup.a
print(tag.attrs)            #输出a标签的属性
print(tag.attrs['class'])   #输出a标签中的class属性
print(tag.attrs['href'])    #输出a标签的链接属性
print(type(tag.attrs))      #输出a属性的类型
print(type(tag))            #输出标签的类型(返回的是一个字典)

结果如下

{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
['py1']
http://www.icourse163.org/course/BIT-268001
<class 'dict'>
<class 'bs4.element.Tag'>

NavigableString元素的处理

import  requests
from bs4 import BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text   #获取到url 地点的源码
soup = BeautifulSoup(demo,'html.parser')
print(soup.a)
print(soup.a.string)    #查办a标签中的字符串
print(soup.p)
print(soup.p.string)    #查看p标签中的字符串
print(type(soup.p.string))  #查看p标签中的字符串的类型

结果如下

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
Basic Python
<p class="title"><b>The demo python introduces several python courses.</b></p>
The demo python introduces several python courses.
<class 'bs4.element.NavigableString'>

关于注释

import  requests
from bs4 import BeautifulSoup

newsoup = BeautifulSoup('<b><!--This is a comment--></b><p>This is not a comment</p>','html.parser')
print(newsoup.b.string)
print(type(newsoup.b.string))
print(newsoup.p.string)
print(type(newsoup.p.string))

结果如下

This is a comment
<class 'bs4.element.Comment'>
This is not a comment
<class 'bs4.element.NavigableString'>

3 基于bs4库的HTML内容遍历方法

回顾demo的例子

import requests
r = requests.get('http://python123.io/ws/demo.html')
demo = r.text
print(demo)

结果如下:

<html><head><title>This is a python demo page</title></head>
<body>
<p class="title"><b>The demo python introduces several python courses.</b></p>
<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>
</body></html>

HTML的基本格式可以分为一个树形结构

标签树的下行遍历

BeautifulSoup类型是标签树的根节点

属性	说明
.contents	子节点的列表，将所有儿子节点存入列表
.children	子节点的迭代类型，与.contents类似，用于循环遍历儿子节点
.descendants	子孙节点的迭代类型，包含所有子孙节点，用于循环遍历

学习代码如下

import requests
from  bs4 import  BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text
soup = BeautifulSoup(demo,'html.parser')
print(soup.head)
print(soup.head.contents)       #.contents返回的类型是列表
print(soup.body.contents)
print(len(soup.body.contents))  #输出body结点的个数
print(soup.body.contents[1])    #输出body结点的第二个

for child in soup.body.children:    #遍历儿子节点
    print(child)

for child in soup.body.descendants:     #遍历子孙节点
	print(child)

输出结果如下

<head><title>This is a python demo page</title></head>
[<title>This is a python demo page</title>]
['\n', <p class="title"><b>The demo python introduces several python courses.</b></p>, '\n', <p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>, '\n']
5
<p class="title"><b>The demo python introduces several python courses.</b></p>


<p class="title"><b>The demo python introduces several python courses.</b></p>


<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>




<p class="title"><b>The demo python introduces several python courses.</b></p>
<b>The demo python introduces several python courses.</b>
The demo python introduces several python courses.


<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>.</p>
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:

<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
Basic Python
 and 
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
Advanced Python
.

标签树的上行遍历

属性	说明
.parent	节点的父亲标签
.parents	节点先辈标签的迭代类型，用于循环遍历先辈节点

学习代码

import requests
from  bs4 import  BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text
soup = BeautifulSoup(demo,'html.parser')

for parent in soup.a.parents:
    if parent is None:      #他的父亲结点可能是空的,所以要加上这样的一句
        print(parent)
    else:
        print(parent.name)

结果如下

p
body
html
[document]

标签树的平行遍历

属性	说明
.next_sibling	返回按照HTML文本顺序的下一个平行节点标签
.previous_sibling	返回按照HTML文本顺序的上一个平行节点标签
.next_siblings	迭代类型，返回按照HTML文本顺序的后续所有平行节点标签
.previous_siblings	迭代类型，返回按照HTML文本顺序的前续所有平行节点标签

平行遍历发生再同一个父结点下的各节点间

import requests
from  bs4 import  BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text
soup = BeautifulSoup(demo,'html.parser')

for sibling in soup.a.next_siblings:  #遍历后续结点
    print(sibling)

for sibling in soup.a.previous_siblings: #遍历
    print(sibling)

结果如下

 and 
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>
.
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:

基于bs4库的HTML格式输出

bs4库的prettify()方法

import requests
from  bs4 import  BeautifulSoup

r = requests.get('http://python123.io/ws/demo.html')
demo = r.text
soup = BeautifulSoup(demo,'html.parser')
soup.prettify()
print(soup.prettify())  #prettify()会为每一个标签添加一个换行符

结果如下:

<html>
 <head>
  <title>
   This is a python demo page
  </title>
 </head>
 <body>
  <p class="title">
   <b>
    The demo python introduces several python courses.
   </b>
  </p>
  <p class="course">
   Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
   <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
    Basic Python
   </a>
   and
   <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
    Advanced Python
   </a>
   .
  </p>
 </body>
</html>

总结:

BeautifulSoup

① 理解Bs4库的基本元素

② 了解遍历的方法