xpath解析库的语法及使用

1、xpath解析库

# Xpath解析库介绍:
	数据解析的过程中使用过正则表达式, 但正则表达式想要进准匹配难度较高, 
	一旦正则表达式书写错误, 匹配的数据也会出错.
    网页由三部分组成: HTML, Css, JavaScript, HTML页面标签存在层级关系,
    即DOM树, 在获取目标数据时可以根据网页层次关系定位标签, 
    再获取标签的文本或属性.

xpath解析库解析数据原理:

1. 根据网页DOM树定位节点标签
2. 获取节点标签的正文文本或属性值

xpath安装, 初体验 --> 使用步骤:

1.xpath安装: pip install lxml
2.requests模块爬取糗事百科热门的标题:
import requests
from lxml import etree

url = 'https://www.qiushibaike.com/'
headers = {
    "User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'
}

res = requests.get(url=url, headers=headers)

tree = etree.HTML(res.text)
title_lst = tree.xpath('//ul/li/div/a/text()')
for item in title_lst:
    print(item)
    
3.xpath使用步骤:
from lxml import etree

tree = etree.HTML(res.text)
tree = etree.parse(res.html, etree.HTMLParse())  # 示例如下, 了解内容

tag_or_attr = tree.xpath('xpath表达式')

xpath解析本地文件

import requests
from lxml import etree

url = 'https://www.qiushibaike.com/'
headers = {
    "User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'
}

res = requests.get(url=url, headers=headers)
with open('qb.html', 'w', encoding='utf-8') as f:
    f.write(res.text)

tree = etree.parse('./qb.html', etree.HTMLParser())
title_lst = tree.xpath('//ul/li/div/a/text()')
for item in title_lst:
    print(item)

2、xpath语法

1.常用规则:
1. nodename: 节点名定位
2. //: 从当前节点选取子孙节点
3. /: 从当前节点选取直接子节点
4. nodename[@attribute="…"] 根据属性定位标签 ‘//div[@class=“ui-main”]’
5. @attributename: 获取属性
6. text(): 获取文本
2.属性匹配两种情况: 多属性匹配 & 单属性多值匹配
2.2 多属性匹配
示例: tree.xpath(’//div[@class=“item” and @name=“test”]/text()’)
2.1 单属性多值匹配
示例: tree.xpath(’//div[contains(@class, “dc”)]/text()’)
3.按序选择:
3.1 索引定位: 从1开始(牢记， 牢记， 牢记)
3.2 last()函数
3.3 position()函数

from lxml import etree

tree = etree.parse('./x.html', etree.HTMLParser())

# 1.根据节点名, 即nodename定位title标签, 获取标签内文字
title_text = tree.xpath('//title/text()')
print(title_text)

# 2.根据节点属性定位: 定位id为007的div标签
div_007 = tree.xpath('//div[@id="007"]')
print(div_007)

# 3.示例直接子节点与子孙节点:/, //
div_007_one = tree.xpath('//div[@id="007"]/text()')
print(div_007_one)
div_007_two = tree.xpath('//div[@id="007"]//text()')
print(div_007_two)

# 4.获取a标签的href属性
a_href = tree.xpath('//div[@class="divtag"]/a/@href')
print(a_href)

# 4.多属性定位: 根据class属性和name属性定位div标签
div_two_attr = tree.xpath('//div[@class="c1" and @name="laoda"]/text()')
print(div_two_attr)

# 5.属性多值定位: 定位所有class中有c1的div标签
div_c1 = tree.xpath('//div[contains(@class, "c1")]')

# 6.按序定位
li_first = tree.xpath('//div[@class="divtag"]/ul/li[1]/text()')  # 定位第一个li标签, 获取其文本
print(li_first)

li_last = tree.xpath('//div[@class="divtag"]/ul/li[last()]/text()')  # 定位最后一个li标签
print(li_last)

li_daotwo = tree.xpath('//div[@class="divtag"]/ul/li[last()-1]/text()')  # 定位倒数第二个li标签
print(li_daotwo)

li_qianthree = tree.xpath('//div[@class="divtag"]/ul/li[position()<4]/text()')  # 定位前三个li标签
print(li_qianthree)

3、案例: 豆瓣电影的相关信息

import requests
from lxml import etree

url = 'https://movie.douban.com/chart'
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

res = requests.get(url=url, headers=headers)
tree = etree.HTML(res.text)
td_list = tree.xpath('//tr[@class="item"]/td[2]')
# print(td_list)
for td in td_list:
    title = td.xpath('./div/a//text()')
    produce = td.xpath('./div/p/text()')
    score = td.xpath('./div/div/span[2]/text()')
    comment_num = td.xpath('./div/div/span[3]/text()')

    print(1111111111111111111111111111111111111111111)
    print(title)
    print(produce)
    print(score)
    print(comment_num)
    print(2222222222222222222222222222222222222222222)