python爬虫之xpath的基本使用

一、简介
XPath 是一门在 XML 文档中查找信息的语言。XPath 可用来在 XML 文档中对元素和属性进行遍历。XPath 是 W3C XSLT 标准的主要元素，并且 XQuery 和 XPointer 都构建于 XPath 表达之上。
二、安装

pip3 install lxml

三、使用

1、导入

from lxml import etree

2、基本使用

from lxml import etree

wb_data = """
        <div>
            <ul>
                 <li class="item-0"><a href="link1.html">first item</a></li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-inactive"><a href="link3.html">third item</a></li>
                 <li class="item-1"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a>
             </ul>
         </div>
        """
html = etree.HTML(wb_data)
print(html)

四、使用案例

1、获取a标签里面的内容

html = etree.HTML(wb_data)
html_data = html.xpath('/html/body/div/ul/li/a')
print(html)
for i in html_data:
    print(i.text)


<Element html at 0x12fe4b8>
first item
second item
third item
fourth item
fifth item

2、获取指定标签里面的内容

html = etree.HTML(wb_data)
html_data = html.xpath('/html/body/div/ul/li/a/text()')
print(html)
for i in html_data:
    print(i)

<Element html at 0x138e4b8>
first item
second item
third item
fourth item
fifth item

3、获取a标签的属性

html = etree.HTML(wb_data)
html_data = html.xpath('/html/body/div/ul/li/a/@href')
for i in html_data:
    print(i)

打印：
link1.html
link2.html
link3.html
link4.html
link5.html

4、获取指定标签里面的内容

html = etree.HTML(wb_data)
html_data = html.xpath('/html/body/div/ul/li/a[@href="link2.html"]/text()')
print(html_data)
for i in html_data:
    print(i)

打印：
['second item']
second item

5、使用li标签下的a标签内容。（使用相对路径）

html = etree.HTML(wb_data)
html_data = html.xpath('//li/a/text()')
print(html_data)
for i in html_data:
    print(i)

打印：
['first item', 'second item', 'third item', 'fourth item', 'fifth item']
first item
second item
third item
fourth item
fifth item

6、查找倒数第二个a标签的href属性

html = etree.HTML(wb_data)
html_data = html.xpath('//li[last()-1]/a/text()')
print(html_data)
for i in html_data:
    print(i)

打印：
['fourth item']
fourth item

7、通过属性名称等提权

//*[@id="kw"]