解析
一、xpath下载与安装
链接:https://pan.baidu.com/s/1TvbRbzFqzuAmr141rP_20w
提取码:dgd1
1.打开chrome浏览器
2.点击右上角小圆点[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-zP8WwfbB-1675093040336)(C:\Users\asonare\AppData\Roaming\Typora\typora-user-images\image-20230114131037054.png)]
3.更多工具
4.扩展程序
5.拖拽xpath文件插件到扩展程序中
6.如果crx文件失效,需要将后缀修改为zip
7.再次拖拽
8.关闭浏览器重新打开
9.ctra+shift+x
10.出现小黑框
二、安装lxml库
#解析本地文件
etree.parse()
#解析服务器响应文件
etree.HTML(response.read().decode('utf-8'))
三、xpath基本用法
1.路径查询
//:查询所有子孙节点
/ :查询子节点
2.谓词查询
//div[@id]
//div[@id=“”]
3.属性查询
//@class
4.模糊查询
//div[contains(@id,“aa”)]
//div[start-with(@id,“aa”)]
5.内容查询
//div/h1/text()
6.逻辑运算
//div[@id=“” and @class=“”]
四、使用xpath获取百度网站的百度一下
import urllib.request
from lxml import etree
url = "https://www.baidu.com/"
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'}
request = urllib.request.Request(url=url,headers=headers)
response = urllib.request.urlopen(request)
content = response.read().decode("utf-8")
tree = etree.HTML(content)
# 获取想要的数据 xpath的返回值时一个列表类型的数据
result = tree.xpath('//input[@id="su"]/@value') # 在chrome浏览器中ctrl+shift+x的小黑框输入 找到百度一下
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-WhxltkMi-1675093040337)(C:\Users\asonare\AppData\Roaming\Typora\typora-user-images\image-20230114152451422.png)]
站长素材的图片下载
def creat_request(page):
if(page ==1):
url = 'https://sc.chinaz.com/tupian/qinglvtupian.html'
else:
url = 'https://sc.chinaz.com/tupian/qinglvtupian_'+str(page)+'.html'
#https://sc.chinaz.com/tupian/qinglvtupian_2.html
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'}
requests = urllib.request.Request(url=url,headers=headers)
return requests
def get_content(request):
response = urllib.request.urlopen(request)
content = response.read().decode('utf-8')
return content
def down_load(content):
tree = etree.HTML(content)
name_list = tree.xpath('//div[@class="container"]//img/@alt')
# 注意 网站的图片一般都会进行懒加载 我们要使用加载前的而不是加载后的
src_list = tree.xpath('//div[@class="container"]//img/@src')
for i in range(len(name_list)):
name = name_list[i]
src = src_list[i]
url = 'https:'+src
urllib.request.urlretrieve(url=url,filename='./photo/' + name + '.jpg')
if __name__ == '__main__':
start_page = int(input("请输入起始页"))
end_page = int(input("请输入结束页"))
for page in range(start_page,end_page+1):
request = creat_request(page) # 请求对象定制
content = get_content(request) # 获取网页源码
down_load(content) # 下载
) # 请求对象定制
content = get_content(request) # 获取网页源码
down_load(content) # 下载