Itm pipeline 为项目管道, 当item生成时,它会自动被送到Item piplone进行处理,常用来:
1,清理HTML数据
2,验证爬取数据,检查爬取字段
3,查看并丢弃重复的内容
4,将爬取的结果保存到数据库
实现Item Pipeline很简单,只需要定义一个类并实现process_item()方法即可,启动Item Pipeline后,Item Pipeline会自动调用这个方法,
process_item()方法必须返回包含数据的字典或Item对象,或者抛出DropItem异常.
例子代码:
爬虫代码:
# -*- coding: utf-8 -*-
import scrapy
from xxoo.items import XxooItem
class MinSpiderSpider(scrapy.Spider):
name = 'min_spider'
allowed_domains = ['quotes.toscrape.com']
start_urls = ['http://quotes.toscrape.com/']
def parse(self, response):
quotes = response.css('.quote')
for i in quotes:
item = XxooItem()
item['text'] = i.css('.text::text').extract_first()
item['author'] = i.css('.author::text').extract_first()
item['tags'] = i.css('.tags .tag::text').extract()
yield item
next1 = response.css('.pager .next a::attr(href)').extract_first()
url = response.urljoin(next1)
yield scrapy.Request(url=url, callback=self.parse)
pipeline代码:
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
from scrapy.exceptions import DropItem
import pymongo
class TextPipeline(object):
def __init__(self):
"""
限制text内容长度为50
"""
self.limit = 50
def process_item(self, item, spider):
"""
先判断item中的text是否存在,不存在抛出异常,存在判断长度,若长度大于50则
截断拼接省略号,再将item返回即可
:param item:
:param spider:
:return:
"""
if item['text']:
if len(item['text']) > self.limit:
item['text'] = item['text'][0:self.limit].rstrip() + '...'
return item
else:
return DropItem('Missing Text')
class MongoPipeline(object):
def __init__(self, mongo_uri, mongo_db):
"""
初始化
"""
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
"""
类方法,用@classmethod标识,是一种依赖注入方法,它的参数就是crawler,通过crawler
我们可以拿到全局配置的每一个配置信息,可以定义mongo_uri,mongo_db链接所需要的地址
和数据库名称,拿到配置信息之后返回类对象即可,这方法主要就是获取settings.py的配置
:param crawler:
:return:
"""
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DB')
)
def open_spider(self, spider):
"""
启动爬虫,调用此方法
:param spider:
:return:
"""
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def process_item(self, item, spider):
"""
执行数据插入操作
:param item:
:param spider:
:return:
"""
name = item.__class__.__name__
self.db[name].insert(dict(item))
return item
def close_spider(self, spider):
"""
当爬虫结束时调用
:param spider:
:return:
"""
self.client.close()
配置settings.py
ITEM_PIPELINES = {
'xxoo.pipelines.TextPipeline': 300,
'xxoo.pipelines.MongoPipeline':400,
}
MONGO_URI = 'localhost'
MONGO_DB = 'xxoo'
赋值ITEM_PIPELINES字典,键名是pipeline的类名称, 键值是调用优先级,是一个数字,数字越小则对应pipeline越先被调用.
执行代码:
scrapy carwl min_spider
最后会在MOngoDB中创建一个叫做xxoo的数据库,QuoteItem的表