免责声明:本文仅做演示与分享~
目录
介绍
scrapy 还提供了处理图片、视频、音频等媒体文件的插件,如:
- scrapy-images:用于下载和处理图片
- scrapy-video:用于下载和处理视频
- scrapy-podcast:用于下载和处理播客
- scrapy-feedstorage:用于存储和处理RSS/Atom订阅源
- scrapy-splash:用于渲染JavaScript渲染后的网页
- scrapy-cdr:用于处理爬取数据并生成CDR(Call Detail Record,呼叫详单记录)
这些插件可以帮助我们更好地处理媒体文件,提高爬虫的效率。
scrapy 还额外提供了一种保存图片的方法:
ImagesPipeline
pip install Pillow # python的图像处理库
站长素材
pipelines.py
from itemadapter import ItemAdapter
import scrapy
from scrapy.pipelines.images import ImagesPipeline
# zz:
class ScrapyDemo1Pipeline(ImagesPipeline): # 继承
# 重写父类方法
def get_media_requests(self, item, info):
# 向图片url发起请求
# meta={"item": item['img_name']} 传递图片的名字.
yield scrapy.Request(item["img_src"], meta={"item": item["img_name"]})
# 指定图片的保存名字:
def file_path(self, request, response=None, info=None, *, item=None):
img_name = request.meta["item"] + ".jpg"
return img_name
items.py
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class ScrapyDemo1Item(scrapy.Item): # 类名可以自定义,但是必须继承scrapy.Item
## zz
img_src = scrapy.Field()
img_name = scrapy.Field()
zz.py
爬虫文件
import scrapy
from scrapy_demo1.items import ScrapyDemo1Item
class ZzSpider(scrapy.Spider):
name = "zz"
# allowed_domains = ["zz.com"]
start_urls = ["https://sc.chinaz.com/tupian/"]
def parse(self, response):
# html 数据
divs = response.xpath('//div[@class="item"]') # 这个地方不能getall(),对象.
# print(len(divs))
for div in divs:
# 图片地址
img_src = div.xpath("./img/@data-original").get()
# 拼接协议
img_src = "https:" + img_src
# 图片的名字
img_name = div.xpath("./img/@alt").get()
print(img_name, img_src)
# https://scpic3.chinaz.net/files/default/imgs/2024-07-31/65ae6ef86d4b1a44
# _s.jpg
# https://scpic3.chinaz.net/files/default/imgs/2024-07-31/65ae6ef86d4b1a44_s.jpg
# break
# # 创建item对象
item = ScrapyDemo1Item()
item["img_src"] = img_src
item["img_name"] = img_name
yield item
settings.py
# 加入配置:保存图片的路径
TMAGES_STORE = "../scrapy_demo1/images"