爬虫学习，xpath练习

最新推荐文章于 2023-09-28 14:37:44 发布

小妖怪_

最新推荐文章于 2023-09-28 14:37:44 发布

阅读量252

点赞数

分类专栏：笔记文章标签： xpath python

本文链接：https://blog.csdn.net/m0_46908895/article/details/106366344

版权

笔记专栏收录该内容

11 篇文章 0 订阅

订阅专栏

from lxml import etree
import requests
import os

dirName = 'Girlslib'
if not os.path.exists(dirName):
    os.mkdir(dirName)

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                  'Chrome/83.0.4103.61 Safari/537.36 '
}
url = 'http://pic.netbian.com/4kmeinv/index_%d.html'

for page in range(1, 6):
    if page == 1:
        new_url = 'http://pic.netbian.com/4kmeinv/'
    else:
        new_url = format(url%page)



    response = requests.get(url=new_url, headers=headers)
    response.encoding = response.apparent_encoding
    page_text = response.text

    #图片名称+图片数据
    tree = etree.HTML(page_text)
    li_list = tree.xpath('//div[@class="slist"]//li')
    for li in li_list:     #    局部数据分析 ./ li的数据类型和tree的数据类型一样，li也可以调用xpath方法

        title = li.xpath('./a/img/@alt')[0] + '.jpg'
        img_src = 'http://pic.netbian.com' + li.xpath('./a/img/@src')[0]
        img_data = requests.get(url=img_src, headers=headers).content
        img_path = dirName + '/' + title
        with open(img_path, 'wb') as fp:
            fp.write(img_data)
        print(title + "  保存成功")