day5-xpath和多线程

最新推荐文章于 2024-07-19 16:36:18 发布

YongQiangA

最新推荐文章于 2024-07-19 16:36:18 发布

阅读量139

点赞数

分类专栏： Python 文章标签： python

本文链接：https://blog.csdn.net/YongQiangA/article/details/119738851

版权

Python 专栏收录该内容

25 篇文章 0 订阅

订阅专栏

xpath和xml数据格式

from lxml import etree

1.专业术语

树：整个html或xml结构
节点：html中的每一个标签，xml中标签就是节点
根节点：树的第一个节点，html的根节点就是html标签
属性：节点属性(html中就是标签属性)

2.xml数据结构

json数据和xml数据是两种通用的数据格式，用于不同语言之间进行数据交换

将一个超市的商品数据进行传输：
json:
{
    "name":"永辉超市",
    "address":"肖家河大厦"
    "goods":[
        {"name":"泡面","price":3.5,"count":20},
        {"name":"矿泉水","price":2,"count":50},
        {"name":"面包","price":5,"count":15},  
    ]
}


xml:
<supermaket>
    <name>永辉超市</name>
    <address>肖家河大厦</address>
    <goodslist>
        <goods name="泡面" price="3.5" count="20"></goods>
        <goods name="矿泉水" price="2" count="50"></goods>
        <goods name="面包" price="5" count="15"></goods>
    </goodslist>
</supermaket>

1)准备数据

xml_str = """
<supermarket>
    <name>永辉超市</name>
    <address>肖家河大厦</address>
    <goodsList>
        <goods name="泡面" price="3.5" count="20"></goods>
        <goods name="矿泉水" price="2" count="50"></goods>
        <goods name="面包" price="5" count="15"></goods>
    </goodsList>
    <worker_list>
        <cashier name="张三" pay="4000"></cashier>
        <shoppingGuide name="李四" pay="3500"></shoppingGuide>
    </worker_list>
    <goods name="烟" price="50" count="15"></goods>
    <goods price="50" count="15">
        <name>烟</name>
    </goods>   
</supermarket>
"""

2)创建数对象，并且获取数据的根节点

supermarket = etree.XML(xml_str)

3)获取标签(获取节点)

节点对象.xpath(路径):根据路径找到对应的节点，返回节点对象，返回保存节点对象的列表
a.写绝对路径：不管xpath前面的节点对象是什么，路径从根节点开始写
写法：/绝对路径

cashier = supermarket.xpath('/supermarket/worker_list/cashier')
print(cashier)  # [<Element cashier at 0x159d8e05ec0>]

work_list = supermarket.xpath('/supermarket/worker_list')[0]
print(work_list)  # <Element worker_list at 0x159d8e05f40>

result = supermarket.xpath('/worker_list/cashier')
print(result)  # []

b.相对路径：
用.来表示当前节点，xpath前面是谁，当前节点就是谁
用…来表示当前节点的上层节点
注意：./可以省略

cashier = supermarket.xpath('./worker_list/cashier')
print(cashier)  # [<Element cashier at 0x2d475693f00>]

cashier = work_list.xpath('./cashier')
print(cashier)  # [<Element cashier at 0x2432e913f00>]

cashier = work_list.xpath('cashier')
print(cashier)  # [<Element cashier at 0x2432e913f00>]

c.//路径：从任意位置开始全局搜索
查找方式和功能和xpath前的节点无关

result = supermarket.xpath('//cashier')
print(result)  # [<Element cashier at 0x1ed5c049180>]

result = supermarket.xpath('//goods')
print(result)  # [<Element goods at 0x25f26b48300>, <Element goods at 0x25f26b48340>, <Element goods at 0x25f26b48380>, <Element goods at 0x25f26b483c0>]

result = supermarket.xpath('//goodsList/goods')
print(result)  # [<Element goods at 0x1e5561a91c0>, <Element goods at 0x1e5561a9200>, <Element goods at 0x1e5561a9240>]

4)获取节点内容

语法：获取节点的路径/text()

name = supermarket.xpath('./name/text()')
print(name)  # ['永辉超市']

name = supermarket.xpath('//name/text()')
print(name)  # ['永辉超市', '烟']

5)获取节点属性值

语法：获取节点的路径/@属性名

result = supermarket.xpath('./goods/@price')
print(result)  # ['50', '50']

result = supermarket.xpath('//goods/@price')
print(result)  # ['3.5', '2', '5', '50', '50']

解析HTML

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
    <h1>永辉超市</h1>
        <p>肖家河大厦</p>
        <p>营业中</p>
        <ul>
            <li>
                <p class="name">泡面</p>
                <p class="price">3.5</p>
                <p class="count">15</p>
            </li>
            <li>
                <p class="name">矿泉水</p>
                <p class="price">2</p>
                <p class="count">120</p>
            </li>
            <li>
                <p class="name">面包</p>
                <p class="price">5</p>
                <p class="count">42</p>
            </li>

            <li>
                <p class="name">充电宝</p>
                <p class="price">150</p>
                <p class="count">10</p>
            </li>

        </ul>
    <div>
        <p class="">p1</p>
        <p class="c1">p2</p>
        <p id="p1">p3</p>
        <p class="c2">p4</p>
    </div>

    <div id="div1">
        <p class="c1">p1</p>
        <p class="p">p2</p>
        <a href="">a1</a>
        <span class="c2">span1</span>
        <img class="c1" src="https://image1.guazistatic.com/92a" alt="">
    </div>

</body>
</html>

from lxml import etree

html = etree.HTML(open('test.html', encoding='utf-8').read())

h1 = html.xpath('/html/body/h1')
print(h1)  # [<Element h1 at 0x212e06c8040>]

h1 = html.xpath('./body/h1')
print(h1)  # [<Element h1 at 0x212e06c8040>]

h1 = html.xpath('//h1')
print(h1)  # [<Element h1 at 0x212e06c8040>]

1.加谓语(加条件)

语法：选中标签的路径[谓语]
1)[N]:获取同层的第N个标签

p = html.xpath('//p/text()')
print(p)  # ['肖家河大厦', '泡面', '3.5', '15', '矿泉水', '2', '120', '面包', '5', '42', '充电宝', '150', '10']
p = html.xpath('//p[1]/text()')
print(p)  # ['肖家河大厦', '泡面', '矿泉水', '面包', '充电宝']

p = html.xpath('./body/p[1]/text()')
print(p)  # ['肖家河大厦']

result = html.xpath('./body/ul/li/p/text()')
print(result)  # ['泡面', '3.5', '15', '矿泉水', '2', '120', '面包', '5', '42', '充电宝', '150', '10']
result = html.xpath('./body/ul/li[1]/p/text()')
print(result)  # ['泡面', '3.5', '15']

result = html.xpath('./body/ul/li/p[last()]/text()')
print(result)  # ['15', '120', '42', '10']

result = html.xpath('./body/ul/li/p[last()-1]/text()')
print(result)  # ['3.5', '2', '5', '150']


result = html.xpath('./body/ul/li[last()-1]/p[last()]/text()')
print(result)  # ['42']

[position()>N]
[position()<N]
[position()>=N]
[position()<=N]

result = html.xpath('./body/ul/li[position()<3]/p/text()')
print(result)  # ['泡面', '3.5', '15', '矿泉水', '2', '120']
result = html.xpath('./body/ul/li[position()<=2]/p/text()')
print(result)  # ['泡面', '3.5', '15', '矿泉水', '2', '120']

result = html.xpath('./body/ul/li[position()>2]/p/text()')
print(result)  # ['面包', '5', '42', '充电宝', '150', '10']

4)[@属性名]：获取有指定属性的标签
p[@class]:有class属性的p标签

result = html.xpath('./body/div/p[@class]/text()')
print(result)  # ['p1', 'p2', 'p4']

[@属性名=属性值]：获取指定属性是指定值的标签

result = html.xpath('./body/div/p[@class="c1"]/text()')
print(result)  # ['p2']

result = html.xpath('//p[@class="c1"]/text()')
print(result)

result = html.xpath('//p[@id="p1"]/text()')
print(result)  # ['p3']

[标签>/</>=/<=/=数据]：将标签按照指定子标签的内容进行筛选

result = html.xpath('./body/ul/li[p[2]>4]/p/text()')
print(result)  # ['面包', '5', '42', '充电宝', '150', '10']

result = html.xpath('./body/ul/li[p[3]>30]/p[1]/text()')
print(result)  # ['矿泉水', '面包']

result = html.xpath('./body/ul/li[p[1]="面包"]/p/text()')
print(result)  # ['面包', '5', '42']

2.通配符:*

1)表示任意标签

result = html.xpath('./body/div[@id="div1"]/*/text()')
print(result)  # ['p1', 'p2', 'a1', 'span1']

result = html.xpath('./body/div[@id="div1"]/*[@class]/text()')
print(result)  # ['p1', 'span1']

2)表示任意属性

result = html.xpath('./body/div[last()]/p[@*]/text()')
print(result)  # ['p1']

result = html.xpath('./body/div[last()]/p[@*="p"]/text()')
print(result)  # ['p2']

result = html.xpath('//img/@*')
print(result)  # ['c1', 'https://image1.guazistatic.com/92a', '']

3.分组(获取若干个路径)😐

# 注意：一个|隔开的必须是两个独立的路径
result = html.xpath('./body/ul/li/p[1]/text()|./body/ul/li/p[3]/text()')
print(result)  # ['泡面', '15', '矿泉水', '120', '面包', '42', '充电宝', '10']

豆瓣电影

from selenium.webdriver import Chrome, ChromeOptions
import csv
from lxml import etree


def get_net_data():
    b = Chrome()
    b.get("https://movie.douban.com/explore#!type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=20&page_start=0")

    html = etree.HTML(b.page_source)
    all_movie = html.xpath('//div[@class="list"]/a')
    all_data = []
    for movie in all_movie:
        img_url = movie.xpath('./div/img/@src')[0]
        name = movie.xpath('./div/img/@alt')[0]
        score = movie.xpath('./p/strong/text()')[0]
        all_data.append([name, score, img_url])

    return all_data


def save_data(data: list):
    writer = csv.writer(open('files/电影.csv', 'w', encoding='utf-8'))
    writer.writerow(['名称', '分数', '封面'])
    writer.writerows(data)


save_data(get_net_data())

YongQiangA

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
day5-xpath和多线程

xpath和xml数据格式from lxml import etree1.专业术语树：整个html或xml结构节点：html中的每一个标签，xml中标签就是节点根节点：树的第一个节点，html的根节点就是html标签属性：节点属性(html中就是标签属性)2.xml数据结构json数据和xml数据是两种通用的数据格式，用于不同语言之间进行数据交换将一个超市的商品数据进行传输：json:{ "name":"永辉超市", "address":"肖家河大厦" "
复制链接

扫一扫