关于使用scrapy框架下载图片掉的坑

最近，公司有个需求，要爬取百度资讯里的文章，且把文章的图片下载到本地，在此期间入坑

首先看SaveImagePipeline管道里代码

#下载文章内容图片管道
class SaveImagePipeline(ImagesPipeline):

    def get_media_requests(self, item, info):
        for every_content_image_url in item["content_image_url"]:
            yield scrapy.Request(every_content_image_url, headers={'Referer': item["content_url"]},
                                 meta={'item': item})
                                 
    def file_path(self, request, response=None, info=None):
        image_name = request.url.split('/')[-1]
        filename = u'{}'.format(folder_strip)
        return filename

运行过后log日志开始不淡定，开始报错，且图片都没有下载下来

接下来我们来看log日志内容

Traceback (most recent call last):
  File "c:\python\python36\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "c:\python\python36\lib\site-packages\scrapy\core\downloader\middleware.py", line 44, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
  File "c:\python\python36\lib\site-packages\twisted\internet\defer.py", line 1362, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200 http://pics7.baidu.com/feed/060828381f30e924b421fb696278e1031c95f752.jpeg?token=dc2e4f093348253c3b3743d802caeb64&s=70311CD5521235CE942C5A3B03005054>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python\python36\lib\site-packages\scrapy\pipelines\files.py", line 419, in media_downloaded
    checksum = self.file_downloaded(response, request, info)
  File "c:\python\python36\lib\site-packages\scrapy\pipelines\images.py", line 106, in file_downloaded
    return self.image_downloaded(response, request, info)
  File "c:\python\python36\lib\site-packages\scrapy\pipelines\images.py", line 118, in image_downloaded
    headers={'Content-Type': 'image/jpeg'})
  File "c:\python\python36\lib\site-packages\scrapy\pipelines\files.py", line 53, in persist_file
    with open(absolute_path, 'wb') as f:
OSError: [Errno 22] Invalid argument: './content_image\\060828381f30e924b421fb696278e1031c95f752.jpeg?token=dc2e4f093348253c3b3743d802caeb64&s=70311CD5521235CE942C5A3B03005054'

开始我还以为是百度具有图片防盗链的功能，然后我就把referer携带上了，但是同样还是报这个错误，最后我看最后一行OSError: [Errno 22] Invalid argument:'./content_image\\060828381f30e924b421fb696278e1031c95f752.jpeg?token=dc2e4f093348253c3b3743d802caeb64&s=70311CD5521235CE942C5A3B03005054'顾明思议：在系统上发生的错误(我使用windows7)然后，我就把图片的名字复制了一下，然后手动在桌面上创建了一张图片保存了一下，我看到下面这个画面

这我幡然顿悟，原来我的问题出现在了60828381f30e924b421fb696278e1031c95f752.jpeg?token=dc2e4f093348253c3b3743d802caeb64&s=70311CD5521235CE942C5A3B03005054`这里的问号，当时我真的感动的想哭~想哭知道吗？

接下来解决

class SaveImagePipeline(ImagesPipeline):

    def get_media_requests(self, item, info):
        for every_content_image_url in item["content_image_url"]:
            yield scrapy.Request(every_content_image_url, headers={'Referer': item["content_url"]},
                                 meta={'item': item})
                                 
    def file_path(self, request, response=None, info=None):
        image_name = request.url.split('/')[-1]
        ########################这是添加的代码###########################
        ***folder_strip = re.sub(r'\?token.+', '', image_name)***
        ########################这是添加的代码###########################
        filename = u'{}'.format(folder_strip)
        return filename

这样顺利去掉?及?后边的,顺利写入本地

博客写的一般，解决的问题也一般，但是我发现网上很少写有这些文章，我相信肯定有和我遇见同样的问题却没有解决的人，所以特地写出来，帮助有需要的人，欢迎各位大佬指出不足，提出整改建议，谢谢