学习爬虫之Scrapy框架学习（六）--1

最新推荐文章于 2024-05-18 17:40:35 发布

面试秘籍Java

最新推荐文章于 2024-05-18 17:40:35 发布

阅读量1.1k

点赞数 8

分类专栏：程序员学习文章标签：学习爬虫 scrapy

本文链接：https://blog.csdn.net/m0_60707606/article/details/136871336

版权

程序员学习专栏收录该内容

8 篇文章 0 订阅

订阅专栏

DEFAULT_IMAGES_URLS_FIELD = ‘image_urls’

DEFAULT_IMAGES_RESULT_FIELD = ‘images’

def init(self, store_uri, download_func=None, settings=None):

super(ImagesPipeline, self).init(store_uri, settings=settings,

download_func=download_func)

解析settings.py里的配置字段

if isinstance(settings, dict) or settings is None:

settings = Settings(settings)

resolve = functools.partial(self._key_for_pipe,

base_class_name=“ImagesPipeline”,

settings=settings)

self.expires = settings.getint(

resolve(“IMAGES_EXPIRES”), self.EXPIRES

)

if not hasattr(self, “IMAGES_RESULT_FIELD”):

self.IMAGES_RESULT_FIELD = self.DEFAULT_IMAGES_RESULT_FIELD

if not hasattr(self, “IMAGES_URLS_FIELD”):

self.IMAGES_URLS_FIELD = self.DEFAULT_IMAGES_URLS_FIELD

self.images_urls_field = settings.get(

resolve(‘IMAGES_URLS_FIELD’),

self.IMAGES_URLS_FIELD

)

self.images_result_field = settings.get(

resolve(‘IMAGES_RESULT_FIELD’),

self.IMAGES_RESULT_FIELD

)

self.min_width = settings.getint(

resolve(‘IMAGES_MIN_WIDTH’), self.MIN_WIDTH

)

self.min_height = settings.getint(

resolve(‘IMAGES_MIN_HEIGHT’), self.MIN_HEIGHT

)

self.thumbs = settings.get(

resolve(‘IMAGES_THUMBS’), self.THUMBS

)

@classmethod

def from_settings(cls, settings):

s3store = cls.STORE_SCHEMES[‘s3’]

s3store.AWS_ACCESS_KEY_ID = settings[‘AWS_ACCESS_KEY_ID’]

s3store.AWS_SECRET_ACCESS_KEY = settings[‘AWS_SECRET_ACCESS_KEY’]

s3store.AWS_ENDPOINT_URL = settings[‘AWS_ENDPOINT_URL’]

s3store.AWS_REGION_NAME = settings[‘AWS_REGION_NAME’]

s3store.AWS_USE_SSL = settings[‘AWS_USE_SSL’]

s3store.AWS_VERIFY = settings[‘AWS_VERIFY’]

s3store.POLICY = settings[‘IMAGES_STORE_S3_ACL’]

gcs_store = cls.STORE_SCHEMES[‘gs’]

gcs_store.GCS_PROJECT_ID = settings[‘GCS_PROJECT_ID’]

gcs_store.POLICY = settings[‘IMAGES_STORE_GCS_ACL’] or None

ftp_store = cls.STORE_SCHEMES[‘ftp’]

ftp_store.FTP_USERNAME = settings[‘FTP_USER’]

ftp_store.FTP_PASSWORD = settings[‘FTP_PASSWORD’]

ftp_store.USE_ACTIVE_MODE = settings.getbool(‘FEED_STORAGE_FTP_ACTIVE’)

store_uri = settings[‘IMAGES_STORE’]

return cls(store_uri, settings=settings)

图片下载

def file_downloaded(self, response, request, info):

return self.image_downloaded(response, request, info)

def image_downloaded(self, response, request, info):

checksum = None

for path, image, buf in self.get_images(response, request, info):

if checksum is None:

buf.seek(0)

checksum = md5sum(buf)

width, height = image.size

self.store.persist_file(

path, buf, info,

meta={‘width’: width, ‘height’: height},

headers={‘Content-Type’: ‘image/jpeg’})

return checksum

图片获取包含了图片大小的过滤，缩略图的生成！

def get_images(self, response, request, info):

path = self.file_path(request, response=response, info=info)

orig_image = Image.open(BytesIO(response.body))

width, height = orig_image.size

if width < self.min_width or height < self.min_height: #图片大小的过滤

raise ImageException(“Image too small (%dx%d < %dx%d)” %

(width, height, self.min_width, self.min_height))

image, buf = self.convert_image(orig_image)

yield path, image, buf

for thumb_id, size in self.thumbs.items(): #缩略图

thumb_path = self.thumb_path(request, thumb_id, response=response, info=info)

thumb_image, thumb_buf = self.convert_image(image, size)

yield thumb_path, thumb_image, thumb_buf

def convert_image(self, image, size=None): #转换成通用格式

if image.format == ‘PNG’ and image.mode == ‘RGBA’:

background = Image.new(‘RGBA’, image.size, (255, 255, 255))

background.paste(image, image)

image = background.convert(‘RGB’)

elif image.mode == ‘P’:

image = image.convert(“RGBA”)

background = Image.new(‘RGBA’, image.size, (255, 255, 255))

background.paste(image, image)

image = background.convert(‘RGB’)

elif image.mode != ‘RGB’:

image = image.convert(‘RGB’)

if size:

image = image.copy()

image.thumbnail(size, Image.ANTIALIAS)

buf = BytesIO()

image.save(buf, ‘JPEG’)

return image, buf

def get_media_requests(self, item, info): #可以用来重写

#将图片的URL变成请求发给引擎

‘’’

req_list=[]

for x in item.get(self.images_urls_field, []): #本句相当于：item[“images_urls”]得到图片URL列表

req_list.append(Request(x))

return req_list

‘’’

return [Request(x) for x in item.get(self.images_urls_field, [])]

def item_completed(self, results, item, info): #此方法获取到了返回的结果（即上面get_media_requests方法的返回值），同时可以获取文件名也可以重写

if isinstance(item, dict) or self.images_result_field in item.fields:

item[self.images_result_field] = [x for ok, x in results if ok]

return item

def file_path(self, request, response=None, info=None):

image_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()

#将url进行hash加密 url是唯一的。所以图片名字是唯一的

return ‘full/%s.jpg’ % (image_guid)

def thumb_path(self, request, thumb_id, response=None, info=None):

#缩略图的存储路径

thumb_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()

return ‘thumbs/%s/%s.jpg’ % (thumb_id, thumb_guid)

（2）咱自己搞个媒体管道进行图片存储（重写框架自带媒体管道类的部分方法实现图片保存名字的自定义！）：

1.spider文件中要拿到图片列表并yield item

2.item里需要定义特殊的字段名：image_urls=scrapy.Field()

3.settings里设置IMAGES_STORE存储路径，如果路径不存在，系统会帮助我们创建

4.使用默认管道则在settings.py文件中开启：scrapy.pipelines.images.ImagesPipeline: 60,

自建管道需要继承ImagesPipeline并在settings.py中开启相应的管道

5.可根据官方文档进行重写：

get_media_requests

item_completed

1.爬虫文件：

-- coding: utf-8 --

import scrapy

import re

from …items import BaiduimgPipeItem

import os

class BdimgSpider(scrapy.Spider):

name = ‘bdimgpipe’

allowed_domains = [‘image.baidu.com’]

start_urls = [‘https://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&sf=1&fmq=&pv=&ic=0&nc=1&z=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&fm=index&pos=history&word=%E7%8C%AB%E5%92%AA’]

def parse(self, response):

text=response.text

image_urls=re.findall(‘“thumbURL”:“(.*?)”’,text)

item=BaiduimgPipeItem()

item[“image_urls”]=image_urls

yield item

2.items.py文件中设置特殊的字段名：

-- coding: utf-8 --

Define here the models for your scraped items

See documentation in:

https://docs.scrapy.org/en/latest/topics/items.html

import scrapy

class BaiduimgPipeItem(scrapy.Item):

define the fields for your item here like:

name = scrapy.Field()

image_urls=scrapy.Field()

3.settings.py文件中开启自建管道并设置文件存储路径：

Configure item pipelines

See https://docs.scrapy.org/en/latest/topics/item-pipeline.html

ITEM_PIPELINES = {

‘baiduimg.pipelines.BaiduimgPipeline’: 300,

‘baiduimg.pipelines.BdImagePipeline’: 40,

‘scrapy.pipelines.images.ImagesPipeline’: 60,

}

IMAGES_STORE =r’C:\my\pycharm_work\爬虫\eight_class\baiduimg\baiduimg\dir0’

IMAGES_STORE =‘C:/my/pycharm_work/爬虫/eight_class_ImagesPipeline/baiduimg/baiduimg/dir3’

4.编写pipelines.py

-- coding: utf-8 --

Define your item pipelines here

Don’t forget to add your pipeline to the ITEM_PIPELINES setting

See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html

from scrapy.http import Request

import os

from scrapy.pipelines.images import ImagesPipeline # 导入要使用的媒体管道类

from .settings import IMAGES_STORE

class BdImagePipeline(ImagesPipeline):

image_num = 0

print(“spider的媒体管道类”)

def get_media_requests(self, item, info): # 可以用来重写

将图片的URL变成请求发给引擎

‘’’

req_list=[]

for x in item.get(self.images_urls_field, []): #本句相当于：item[“images_urls”]得到图片URL列表

req_list.append(Request(x))

return req_list

‘’’

return [Request(x) for x in item.get(self.images_urls_field, [])]

def item_completed(self, results, item, info):

images_path = [x[“path”] for ok,x in results if ok]

for image_path in images_path: # 通过os的方法rename实现图片保存名字的自定义！第一个参数为图片原路径；第二个参数为图片自定义路径

os.rename(IMAGES_STORE+“/”+image_path,IMAGES_STORE+“/”+str(self.image_num)+“.jpg”) # IMAGES_STORE+“/”+image_path是图片保存的原绝对路径；第二个参数是自定义的图片保存的新绝对路径（此处也放在IMAGES_STORE路径下）

self.image_num+=1

‘’’

源码中一个可重写的方法：

def item_completed(self, results, item, info): #此方法也可以重写

if isinstance(item, dict) or self.images_result_field in item.fields:

item[self.images_result_field] = [x for ok, x in results if ok]

return item

results详解：

url-从中下载文件的网址。这是从get_media_requests() 方法返回的请求的URL 。

path- FILES_STORE文件存储的路径（相对于）

checksum- 图片内容的MD5哈希

这是该results参数的典型值：

[(True,

{‘checksum’: ‘2b00042f7481c7b056c4b410d28f33cf’,

‘path’: ‘full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg’,

‘url’: ‘http://www.example.com/files/product1.pdf’}),

]

而上面的方法item_completed()就是处理此results的，所以解读源码：

[x for ok, x in results if ok]

可知此列表推导式获取的是results中整个字典的值，然后赋给item再返回！

依此得出思路，可通过下面列表推导式获取results中图片存储的路径：

images_path = [x[“path”] for ok,x in results if ok]

‘’’

5.观察可发现完美实现：

在这里插入图片描述

3.如果你感觉就这三十张可爱的猫咪的图片不够，那咱就把爬虫文件改下，想要多少就要多少！！！注意：settings.py文件中设置个下载延迟哦！不然会有被封的危险。

-- coding: utf-8 --

import scrapy

import re

from …items import BaiduimgItem,BaiduimgPipeItem

import os

class BdimgSpider(scrapy.Spider):

name = ‘bdimgpipe’

allowed_domains = [‘image.baidu.com’]

page_url=“https://image.baidu.com/search/acjson?tn=resultjson_com&ipn=rj&ct=201326592&is=&fp=result&queryWord=%E7%8C%AB%E5%92%AA&cl=2&lm=-1&ie=utf-8&oe=utf-8&adpicid=&st=-1&z=&ic=0&hd=&latest=&copyright=&word=%E7%8C%AB%E5%92%AA&s=&se=&tab=&width=&height=&face=0&istype=2&qc=&nc=1&fr=&expermode=&force=&pn={}&rn=30&gsm=1e&1588088573059=”

num = 0

pagenum=1

def parse(self, response):

text=response.text

image_urls=re.findall(‘“thumbURL”:“(.*?)”’,text)

item=BaiduimgPipeItem()

做了那么多年开发，自学了很多门编程语言，我很明白学习资源对于学一门新语言的重要性，这些年也收藏了不少的Python干货，对我来说这些东西确实已经用不到了，但对于准备自学Python的人来说，或许它就是一个宝藏，可以给你省去很多的时间和精力。

别在网上瞎学了，我最近也做了一些资源的更新，只要你是我的粉丝，这期福利你都可拿走。

我先来介绍一下这些东西怎么用，文末抱走。

（1）Python所有方向的学习路线（新版）

这是我花了几天的时间去把Python所有方向的技术点做的整理，形成各个领域的知识点汇总，它的用处就在于，你可以按照上面的知识点去找对应的学习资源，保证自己学得较为全面。

最近我才对这些路线做了一下新的更新，知识体系更全面了。