scrapy图片数据爬取之ImagesPipeline

前言

• 基于scrapy爬取字符串类型的数据和爬取图片类型的数据区别？

  • 字符串：只需要基于xpath进行解析且提交管道进行持久化存储

  • 图片：xpath解析出图片src的属性值。单独的对图片地址发起请求获取图片二进制类型的数据

• ImagesPipeline：

  • 只需要将img的src的属性值进行解析，提交到管道，管道就会对图片的src进行请求发送获取图片的二进制类型的数据，且还会帮我们进行持久化存储。

• 需求：爬取站长素材中的高清图片

实现代码

生成名为img的spider文件，在img文件中解析图片地址。

注意：此处图片地址src使用了伪属性，就是当使用抓包工具进行定位时，可视化界面图片的地址为src，不可视化界面为src2，因此我们直接定位src时会报错None，也就是获取不到内容。所以应该定位到src2。

在items文件中定义一个 ImgsproItem类，用于存储图片地址。

随后在spider文件中从imgsPro.items导入 ImgsproItem，并实例化一个item对象。

最后将item提交管道。

sipder文件

 #需求：爬取站长素材中的高清图片
 import scrapy
 from imgsPro.items import ImgsproItem
 ​
 ​
 class ImgSpider(scrapy.Spider):
     name = 'img'
     #allowed_domains = ['www.xxx.com']
     start_urls = ['https://sc.chinaz.com/tupian/']
 ​
     def parse(self, response):
         div_list=response.xpath('//div[@id="container"]/div')
         for div in div_list:
             #注意使用伪属性src2
             list1=[]
             src=div.xpath('./div/a/img/@src2').extract_first()
             src='http'+src
             list1.append(src)
             item=ImgsproItem()
             item['src']=list1
             yield item

items文件

 # Define here the models for your scraped items
 #
 # See documentation in:
 # https://docs.scrapy.org/en/latest/topics/items.html
 ​
 import scrapy
 ​
 ​
 class ImgsproItem(scrapy.Item):
     # define the fields for your item here like:
     src = scrapy.Field()
     pass

在pipeline文件中自定制一个基于ImagesPipeLine的一个管道类

• get_media_request（请求）

• file_path（指定路径）

• item_completed（返回下一个即将被执行的管道类）

  指定图片名称imgName时，使用切分，取图片路径的最后一个元素为图片名称。

   import scrapy
   from itemadapter import ItemAdapter
   ​
   ​
   # class ImgsproPipeline:
   #     def process_item(self, item, spider):
   #         return item
   from scrapy.pipelines.images import ImagesPipeline
   class imgsPipeLine(ImagesPipeline):
   ​
       #根据图片地址进行图片数据的请求
       def get_media_requests(self, item, info):
           yield scrapy.Request(item['src'])
   ​
       #指定图片存储的路径
       def file_path(self, request, response=None, info=None, *, item=None):
           imgName=request.url.split('/')[-1]
           return imgName
   ​
       def item_completed(self, results, item, info):
           return item#返回给下一个即将被执行的管道类

在setting文件中，设定UA伪装，指定LOG_ERROR='ERROR'仅输出错误日志，不遵守君子协定，修改为正确的ITEM_PIPELINES，最后指定文件寸尺的目录。

 # Scrapy settings for imgsPro project
 #
 # For simplicity, this file contains only settings considered important or
 # commonly used. You can find more settings consulting the documentation:
 #
 #     https://docs.scrapy.org/en/latest/topics/settings.html
 #     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
 #     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
 ​
 BOT_NAME = 'imgsPro'
 ​
 SPIDER_MODULES = ['imgsPro.spiders']
 NEWSPIDER_MODULE = 'imgsPro.spiders'
 ​
 ​
 # Crawl responsibly by identifying yourself (and your website) on the user-agent
 USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3877.400 QQBrowser/10.8.4508.400'
 LOG_ERROR='ERROR'
 # Obey robots.txt rules
 ROBOTSTXT_OBEY = False
 ​
 # Configure maximum concurrent requests performed by Scrapy (default: 16)
 #CONCURRENT_REQUESTS = 32
 ​
 # Configure a delay for requests for the same website (default: 0)
 # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
 # See also autothrottle settings and docs
 #DOWNLOAD_DELAY = 3
 # The download delay setting will honor only one of:
 #CONCURRENT_REQUESTS_PER_DOMAIN = 16
 #CONCURRENT_REQUESTS_PER_IP = 16
 ​
 # Disable cookies (enabled by default)
 #COOKIES_ENABLED = False
 ​
 # Disable Telnet Console (enabled by default)
 #TELNETCONSOLE_ENABLED = False
 ​
 # Override the default request headers:
 #DEFAULT_REQUEST_HEADERS = {
 #   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 #   'Accept-Language': 'en',
 #}
 ​
 # Enable or disable spider middlewares
 # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
 #SPIDER_MIDDLEWARES = {
 #    'imgsPro.middlewares.ImgsproSpiderMiddleware': 543,
 #}
 ​
 # Enable or disable downloader middlewares
 # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
 #DOWNLOADER_MIDDLEWARES = {
 #    'imgsPro.middlewares.ImgsproDownloaderMiddleware': 543,
 #}
 ​
 # Enable or disable extensions
 # See https://docs.scrapy.org/en/latest/topics/extensions.html
 #EXTENSIONS = {
 #    'scrapy.extensions.telnet.TelnetConsole': None,
 #}
 ​
 # Configure item pipelines
 # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
 ITEM_PIPELINES = {
     'imgsPro.pipelines.imgsPipeLine': 300,
 }
 ​
 # Enable and configure the AutoThrottle extension (disabled by default)
 # See https://docs.scrapy.org/en/latest/topics/autothrottle.html
 #AUTOTHROTTLE_ENABLED = True
 # The initial download delay
 #AUTOTHROTTLE_START_DELAY = 5
 # The maximum download delay to be set in case of high latencies
 #AUTOTHROTTLE_MAX_DELAY = 60
 # The average number of requests Scrapy should be sending in parallel to
 # each remote server
 #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
 # Enable showing throttling stats for every response received:
 #AUTOTHROTTLE_DEBUG = False
 ​
 # Enable and configure HTTP caching (disabled by default)
 # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
 #HTTPCACHE_ENABLED = True
 #HTTPCACHE_EXPIRATION_SECS = 0
 #HTTPCACHE_DIR = 'httpcache'
 #HTTPCACHE_IGNORE_HTTP_CODES = []
 #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
 ​
 MEDIA_ALLOW_REDIRECTS=True
 ​
 #指定图片存储的目录
 IMAGES_STORE='./imgs_dengdeng'

接下来直接在终端输入scrapy crawl img即可！