scrapy 图片爬取过程简单但是坎坷pillow

直接代码

import scrapy
from imgProject.items import ImgprojectItem

class ImgfirstSpider(scrapy.Spider):
    name = 'imgfirst'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['https://sc.chinaz.com/tupian/']

    def parse(self, response):
        div_list = response.xpath('//*[@id="container"]/div')
        for div in div_list:
            src = 'https:' + div.xpath('./div/a/img/@src2').extract_first()
            print(src)
            item = ImgprojectItem()
            item['src'] = src
            yield item
class ImgprojectItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    src = scrapy.Field()
ROBOTSTXT_OBEY = False

LOG_LEVEL = 'ERROR'

USER_AGENT = 请求中复制

ITEM_PIPELINES = {
   # 'imgProject.pipelines.ImgprojectPipeline': 300,
   'imgProject.pipelines.imgsPipeLine': 300,
}

IMAGES_STORE = './imgs'

import scrapy
from scrapy.pipelines.images import ImagesPipeline


# class ImgprojectPipeline:
#     def process_item(self, item, spider):
#         return item


class imgsPipeLine(ImagesPipeline):

    def get_media_requests(self, item, info):
        print('get_media_request')
        yield scrapy.Request(item['src'])

    def file_path(self, request, response=None, info=None):
        imgName = request.url.split('/')[-1]
        print('file_path')
        return imgName

    def item_completed(self, results, item, info):
        print('item_completed')
        return item

小结: URL没有问题就是不能下载图片

代码没有问题看了网上好多解答,终于看到大佬的解答,人瞬间蛋疼起来

就是没有下载标题上的哪个库,

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

xiaodunmeng

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值