day5-xpath和多线程

day5-xpath和多线程

1.xpath 和 xml 数据格式

(1) 专业术语

• 树: 整个 html 或 xml 结构
• 节点: html 中的每个标签，xml 中标签就是节点
• 根节点: 树的第一个节点，html 的根节点就是 html 标签
• 属性: 节点属性 (html 中就是标签属性)

(2) xml 数据格式

• json 数据和 xml 数据是两种通用的数据格式，用于不同语言之间进行数据交流

将一个超市的商品数据进行传输：
json:
{
    "name": "永辉超市",
    "address": "肖家河大厦",
    "goods": [
        {"name": "泡面", "price": 3.5, "count": 20},
        {"name": "矿泉水", "prcie": 2, "count": 50},
        {"name": "面包", "price": 5, "count": 15}
    ]
}

xml:
<supermarket>
    <name>永辉超市</name>
    <address>肖家河大厦</address>
    <goodsList>
        <goods name="泡面" price="3.5" count="20"></goods>
        <goods name="矿泉水" price="2" count="50"></goods>
        <goods name="面包" price="5" count="15"></goods>
    </goods>
</supermarket>

(3) 准备数据

xml_str = """
<supermarket>
    <name>永辉超市</name>
    <address>肖家河大厦</address>
    <goodsList>
        <goods name="泡面" price="3.5" count="20"></goods>
        <goods name="矿泉水" price="2" count="50"></goods>
        <goods name="面包" price="5" count="15"></goods>
    </goodsList>
    <worker_list>
        <cashier name="张三" pay="4000"></cashier>
        <shoppingGuide name="李四" pay="3500"></shoppingGuide>
    </worker_list>
    <goods price="50" count="15">
         <name>烟</name>
    </goods>

</supermarket>
"""

(4) 创建树对象，并且获取数据的根节点

supermarket = etree.XML(xml_str)
print(supermarket)

(5) 获取标签(获取节点)

• 节点对象.xpath(路径) - 根据路径找到对应的节点，返回保存节点对象的列表
• a. 写绝对路径: 不管 xpath 前面的节点对象是什么，路径从根节点开始写
• 写法: /绝对路径

cashier = supermarket.xpath('/supermarket/worker_list/cashier')
print(cashier)

worker_list = supermarket.xpath('/supermarket/worket_list')[0]
print(worket_list)

result = worket_list.xpath('/worket_list/cashier')
print(result)

• 相对路径 : 用 . 来表示当前节点，xpath 前面是谁，当前节点就是谁

  ​ : 用 … 来表示当前节点的上层节点

• 注意: ./ 可以省略

cashier = supermarket.xpath('./worker_list/cashier')
print(cashier)

cashier = worker_list.xpath('./cashier')
print(cashier)

cashier = supermarket.xpath('worker_list/cashier')
print(cashier)

cashier = worker_list.xpath('cashier')
print(cashier)

• // 路径 - 从任意位置开始全局搜索
• 查找方向和功能的 xpath 前的节点无关

result = supermarket.xpath('//cashier')
print(result)

result = supermarket.xpath('//godds')
print(result)

result = supermarket.xpath('//goodsList/goods')
print(result)

(6) 获取节点内容

• 语法: 获取节点的路径 /text()

name = sueprmarket.xpath('./name/text()')
print(name)

names = supermarket.xpath('//name/text()')
print(names)

(7) 获取节点属性值

• 语法: 获取节点的路径 /@属性名

result = supermarket.xpath('./goods/@price')
print(result)

result = supermarket.xpath('//goods/@price')
print(result)

2.解析 html

from lxml import etree

html = etree.HTML(open('test.html', encoding='utf-8').read())

h1 = html.xpath('/html/body/h1')
print(h1)

h1 = html.xpath('./body/h1')
print(h1)

h1 = html.xpath('//h1')
print(h1)

(1) 加谓语 (加条件)

• 语法: 选中标签的路径[谓语]
• [N] - 获取同层的第N个标签

p = html.xpath('./body/p[1]/text()')
print(p)

result = html.xpath('./body/ul/li[last()-1]/p[last()]/text()')
print(result)

# [position()>N]
# [position()<N]
# [position()>=N]
# [position()<=N]

result = html.xpath('./body/ul/li[position()<=2]/p/text()')
print(result)

result = html.xpath('./body/ul/li[position()>2]/p/text()')
print(result)

• [@属性名=属性值] - 获取指定属性是指定值的标签
• p[@class] - 有 class 属性的 p 标签

result = html.xpath('./body/div/p[@class]/text()')
print(result)

• [@属性名=属性值] - 获取指定属性是指定值的标签

result = html.xpath('./body/div/p[@class="c1"]/text()')
print(result)

result = html.xpath('//p[@class="c1"]/text()')
print(result)

result = html.xpath('./body/div/p[@id="p1"]/text()')
print(result)

• [标签 >/</>=/<=/= 数据] - 将标签按照指定子标签的内容进行筛选

result = html.xpath('./body/ul/li[p[2]>4]/p/text()')
print(result)

result = html.xpath('./body/ul/li[p[3]>30]/p[1]/text()')
print(result)


result = html.xpath('./body/ul/li[p[1]="面包"]/p/text()')
print(result)

(2) 通配符: *

• 表示任意标签

result = html.xpath('./body/div[@id="div1"]/*')
print(result)

result = html.xpath('./body/div[@id="div1"]/*[@class]')
print(result)

result = html.xpath('//*[@class="c1"]')
print(result)

• 表示任意属性

result = html.xpath('./body/div[last()]/p[@*]/text()')
print(result)

result = html.xpath('./body/div[last()]/p[@*="p"]/text()')
print(result)

result = html.xpath('//img/@*')
print(result)

(3) 分支 (获取若干个路径) - |

• 注意: 一个 | 隔开的必须是两个独立的路径

result = html.xpath('./body/ul/li/p[1]/text()|./body/ul/li/p[3]/text()')
print(result)

3.豆瓣电影

from selenium.webdriver import Chrome, ChromeOptions
import csv
from lxml import etree

def get_net_data():
    b = Chrome()
    b.get("https://movie.douban.com/explore#!type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=20&page_start=0")
    
    html = etree.HTML(b.page_source)
    all_movie = html.xpath('//div[@class="list"]/a')
    all_data = []
    for movie in all_movie:
        img_url = movie.xpath('./div/img/@src')[0]
        name = movie.xpath('./div/img/@alt')[0]
        score = movie.xpath('./p/strong/text()')[0]
        all_data.append([name, score, img_url])

    return all_data


def save_data(data: list):
    writer = csv.writer(open('files/电影分类.csv', 'w', encoding='utf-8'))
    writer.writerow(['名称', '分数', '封面'])
    writer.writerows(data)


save_data(get_net_data())