python解析库 爬虫_python 爬虫（二）- 解析库 的简单使用

当我们在获取到网页相应内容的时候， 就会使用去解析它 过滤得到想要的内容

正则re

lxml 库

Beautiful Soup

pyquery

JsonPath

示例响应内容

http://quotes.toscrape.com/ 截取部分内容， 以下所有例子将以这个响应内容来示范， 假设响应的内容字符串 定义为一个变量 content

一、正则re

使用python 中内置的模块 re正则模块

如解析页面 上所有的名人的名字：

import re

pat = re.compile('(.*?)')

print(pat.findall(content))

输出：['Albert Einstein', 'J.K. Rowling', 'Albert Einstein', 'Jane Austen', 'Marilyn Monroe', 'Albert Einstein', 'André Gide', 'Thomas A. Edison', 'Eleanor Roosevelt', 'Steve Martin']

二、lxml库

lxml 支持xpath 的解析方式，那什么是xpath解析呢？

XPath 使用路径表达式来选取 XML 文档中的节点或节点集。节点是通过沿着路径 (path) 或者步 (steps) 来选取的。 xpath 解析方式

同样使用上面的例子，首先需要安装 lxml库

from lxml import etree

html = etree.HTML(content)

authors = html.xpath("//small[@class='author']//text()")

print(authors)

三、Beautiful Soup

BeautifulSoup也是Python的一个HTML或XML解析库，最主要的功能就是从网页爬取我们需要的数据。

首先需要安装 BeautifulSoup 解析器 pip install beautifulsoup4

from bs4 import BeautifulSoup

soup = BeautifulSoup(content,"lxml")

authors = soup.select('small.author')

for author in authors:

print(author.get_text())

四、pyquery

pyquery语法与前端 jQuery的用法几乎一样

from pyquery import PyQuery as pq

doc = pq(content)

authors = doc('small.author')

for author in authors.items():

print(author.text())

会使用jsonpath的地方 ， 一般响应的内容 是json数据 。

语法：

XPath

JSONPath

Result

/store/book/author

$.store.book[*].author

the authors of all books in the store

//author

$..author

all authors

/store/*

$.store.*

all things in store, which are some books and a red bicycle.

/store//price

$.store..price

the price of everything in the store.

//book[3]

$..book[2]

the third book

//book[last()]

$..book[(@.length-1)] $..book[-1:]

the last book in order.

//book[position()<3]

$..book[0,1] $..book[:2]

the first two books

//book[isbn]

$..book[?(@.isbn)]

filter all books with isbn number

//book[price<10]

$..book[?(@.price<10)]

filter all books cheapier than 10

//*

$..*

all Elements in XML document. All members of JSON structure.

这里使用 一段 json 数据

我们来获取 所有的作者 和 所有价格

import jsonpath

import json

json_str = '''

{ "store": {

"book": [

{ "category": "reference",

"author": "Nigel Rees",

"title": "Sayings of the Century",

"price": 8.95

},

{ "category": "fiction",

"author": "Evelyn Waugh",

"title": "Sword of Honour",

"price": 12.99

},

{ "category": "fiction",

"author": "Herman Melville",

"title": "Moby Dick",

"isbn": "0-553-21311-3",

"price": 8.99

},

{ "category": "fiction",

"author": "J. R. R. Tolkien",

"title": "The Lord of the Rings",

"isbn": "0-395-19395-8",

"price": 22.99

}

],

"bicycle": {

"color": "red",

"price": 19.95

}

}

}

'''

jc = json.loads(json_str)

jp = jsonpath.jsonpath(jc, '$..author')

print(jp)

jp = jsonpath.jsonpath(jc, '$.store..price')

print(jp)

输出：

['Nigel Rees', 'Evelyn Waugh', 'Herman Melville', 'J. R. R. Tolkien']

[8.95, 12.99, 8.99, 22.99, 19.95]