Python_Xpath使用

最新推荐文章于 2020-11-01 12:10:18 发布

十一姐

最新推荐文章于 2020-11-01 12:10:18 发布

阅读量330

点赞数 1

分类专栏： # SpiderCrawl

本文链接：https://blog.csdn.net/weixin_43411585/article/details/88929974

版权

SpiderCrawl 专栏收录该内容

47 篇文章 261 订阅

订阅专栏

表达式	描述
nodename	选择此节点的所有子节点，tag或者*选择任意的tag
/	从根节点选取，从当前节点选取直接子节点，不包含孙子
//	从当前节点选取子孙节点
。	选取当前节点
。。	选取当前节点的父节点
@	选取属性

from lxml import etree
text = '''
<div>
    <ul>
         <li class="item-0"><a href="link1.html">first item</a></li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-inactive"><a href="link3.html">third item</a></li>
         <li class="item-1"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a>
     </ul>
 </div>
'''
# html = etree.parse('.test.html',etree.HTMLParser())
html = etree.HTML(text)
result = etree.tostring(html)
print(result.decode('utf-8'))
# 运行结果
# <html><body><div>
#     <ul>
#          <li class="item-0"><a href="link1.html">first item</a></li>
#          <li class="item-1"><a href="link2.html">second item</a></li>
#          <li class="item-inactive"><a href="link3.html">third item</a></li>
#          <li class="item-1"><a href="link4.html">fourth item</a></li>
#          <li class="item-0"><a href="link5.html">fifth item</a>
#      </li></ul>
#  </div>
# </body></html>

4、节点匹配实例

.xpath(’//*’)：代表匹配所有节点，返回类型为列表，列表里的每个元素是Element类型
.xpath(’//li’)：代表匹配所有li节点
.xpath(’//li/a’)：代表匹配li节点的所有子节点a
.xpath(’//a[@href=“link4.html”]/…/@class’)：代表匹配href属性为link4.html的a节点，然后再获取其父节点，然后再获取其class属性
.xpath(’//li[@class=“item-0”]’)：代表匹配节点class属性为item-0
.xpath(’//li[@class=“item-0”]/a/text()’)：代表匹配a节点的文本
.xpath(’//li/a/@href’)：获取a节点的href属性

from lxml import etree
text = '''
<div>
    <ul>
         <li class="item-0"><a href="link1.html">first item</a></li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-inactive"><a href="link3.html">third item</a></li>
         <li class="item-1"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a>
     </ul>
 </div>
'''
html = etree.HTML(text)
result = html.xpath('//*')
print(result)  # [<Element html at 0x2893a48>, <Element body at 0x28939c8>, <Element div at 0x2893988>, <Element ul at 0x2893a88>, <Element li at 0x2893ac8>, <Element a at 0x2893b48>, <Element li at 0x2893b88>, <Element a at 0x2893bc8>, <Element li at 0x2893c08>, <Element a at 0x2893b08>, <Element li at 0x2893c48>, <Element a at 0x2893c88>, <Element li at 0x2893cc8>, <Element a at 0x2893d08>]
result = html.xpath('//li')
print(result)  # [<Element li at 0x2894ac8>, <Element li at 0x2894a88>, <Element li at 0x2894b88>, <Element li at 0x2894bc8>, <Element li at 0x2894c08>]
print(result[0])  # <Element li at 0x2894ac8>
result = html.xpath('//li/a')
print(result)  # [<Element a at 0x2894a48>, <Element a at 0x2894a08>, <Element a at 0x2894b08>, <Element a at 0x2894b48>, <Element a at 0x2894b88>]
result = html.xpath('//a[@href="link4.html"]/../@class')
print(result)  # ['item-1']
result = html.xpath('//li[@class="item-0"]')
print(result)  # [<Element li at 0x28939c8>, <Element li at 0x2893988>]
result = html.xpath('//li[@class="item-0"]/a/text()')
print(result)  # ['first item', 'fifth item']
result = html.xpath('//li[@class="item-0"]//text()')
print(result)  # ['first item', 'fifth item', '\n     ']
result = html.xpath('//li/a/@href')
print(result)  # ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']

4、属性多值匹配

.xpath(’//li[contains(@class,“li”)]/a/text()’)：属性多值匹配，通过contains()方法，第一个参数传入属性名称，第二个参数传入属性值，只要此属性包含所传入的属性值，就可以完成匹配了

from lxml import etree
text = '''
<li class="li li-first"><a href="link.html">first item</a></li>
'''
html = etree.HTML(text)
result = html.xpath('//li[contains(@class,"li")]/a/text()')
print(result)  # ['first item']

5、多属性匹配

.xpath(’//li[contains(@class,“li”) and @name=“item”]/a/text()’)：根据多个属性确定一个节点，用and连接

from lxml import etree
text = '''
<li class="li li-first" name="item"><a href="link.html">first item</a></li>
'''
html = etree.HTML(text)
result = html.xpath('//li[contains(@class,"li") and @name="item"]/a/text()')
print(result)  # ['first item']

运算符及其介绍

运算符	描述	实例	返回值
or	或	age=19 or age=20	true,false
and	与	age>19 and age<21	true,false
mod	余数	5 mod 2	1
l	计算两个节点集	//book l //cd	返回所有拥有book和cd的节点集
+	加法	6 + 2	7
-	减法	5 -2	3
*	乘法	5 * 2	10
div	除法	5 / 2	2
=	等于	age=19	true,false
!=	不等于	age!=19	true,false
<	小于	age<19	true,false
<=	小于等于	age<=19	true,false
>	大于	age>19	true,false
>=	大于等于	age>=19	true,false

6、按序选择

.xpath(’//li[1]/a/text()’)：选取第一个li节点，中括号中传入数字1即可。注意，这里和代码中不同，序号是以1开头的，不是以0开头的
.xpath(’//li[last()]/a/text()’)：选取最后一个li节点，中括号中传入last()即可
.xpath(’//li[position()< 3]/a/text()’)：选择位置小于3的li节点
.xpath(’//li[last()-2]/a/text()’)：选取了倒数第三个li节点
http://www.w3school.com.cn/xpath/xpath_functions.asp

from lxml import etree
text = '''
<div>
    <ul>
         <li class="item-0"><a href="link1.html">first item</a></li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-inactive"><a href="link3.html">third item</a></li>
         <li class="item-1"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a>
     </ul>
 </div>
'''
html = etree.HTML(text)
result = html.xpath('//li[1]/a/text()')
print(result)  # ['first item']
result = html.xpath('//li[last()]/a/text()')
print(result)  # ['fifth item']
result = html.xpath('//li[position()<3]/a/text()')
print(result)  # ['first item', 'second item']
result = html.xpath('//li[last()-2]/a/text()')
print(result)  # ['third item']

7、节点轴选择

XPath提供了很多节点轴选择方法，包括获取子元素，兄弟元素，父元素，祖元素等
http://www.w3school.com.cn/xpath/xpath_axes.asp

from lxml import etree
text = '''
<div>
    <ul>
         <li class="item-0"><a href="link1.html"><span>first item</span></a></li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-inactive"><a href="link3.html">third item</a></li>
         <li class="item-1"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a>
     </ul>
 </div>
'''
html = etree.HTML(text)
result = html.xpath('//li[1]/ancestor::*')
print(result)  # 调用ancestor轴，获取所有祖先节点
result = html.xpath('//li[1]/ancestor::div')
print(result)  # 获取只有div的祖先节点
result = html.xpath('//li[1]/attribute::*')
print(result)  # 调用attribute轴，获取所有属性值
result = html.xpath('//li[1]/child::a[@href="link1.html"]')
print(result)  # 调用child轴获取所有直接子节点
result = html.xpath('//li[1]/descendant::span')
print(result)  # 调用descendant轴，获取所有子孙节点
result = html.xpath('//li[1]/following::*[2]')
print(result)  # 调用following轴，获取当前节点之后的所有节点，这里我们虽然使用的是*匹配，但又加了索引选择，所以只获取了第二个后续节点
result = html.xpath('//li[1]/following-sibling::*')
print(result)  # 调用following-sibling轴，获取当前节点之后的所有统计节点

# 运行结果
# [<Element html at 0x2893b88>, <Element body at 0x2893b08>, <Element div at 0x2893ac8>, <Element ul at 0x2893bc8>]
# [<Element div at 0x2893ac8>]
# ['item-0']
# [<Element a at 0x2893bc8>]
# [<Element span at 0x2893ac8>]
# [<Element a at 0x2893bc8>]
# [<Element li at 0x2893b08>, <Element li at 0x2893c08>, <Element li at 0x2893c48>, <Element li at 0x2893c88>]