scrapy 中的pipeleine.py文件解答

Itm pipeline 为项目管道, 当item生成时,它会自动被送到Item piplone进行处理,常用来:

1,清理HTML数据

2,验证爬取数据,检查爬取字段

3,查看并丢弃重复的内容

4,将爬取的结果保存到数据库

实现Item Pipeline很简单,只需要定义一个类并实现process_item()方法即可,启动Item Pipeline后,Item Pipeline会自动调用这个方法,

process_item()方法必须返回包含数据的字典或Item对象,或者抛出DropItem异常.

例子代码: 

爬虫代码:

# -*- coding: utf-8 -*-
import scrapy

from xxoo.items import XxooItem


class MinSpiderSpider(scrapy.Spider):
    name = 'min_spider'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):
        quotes = response.css('.quote')
        for i in quotes:
            item = XxooItem()
            item['text'] = i.css('.text::text').extract_first()
            item['author'] = i.css('.author::text').extract_first()
            item['tags'] = i.css('.tags .tag::text').extract()

            yield item

        next1 = response.css('.pager .next a::attr(href)').extract_first()
        url = response.urljoin(next1)
        yield scrapy.Request(url=url, callback=self.parse)

 pipeline代码:

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
from scrapy.exceptions import DropItem
import pymongo


class TextPipeline(object):
    def __init__(self):
        """
        限制text内容长度为50
        """
        self.limit = 50

    def process_item(self, item, spider):
        """
        先判断item中的text是否存在,不存在抛出异常,存在判断长度,若长度大于50则
        截断拼接省略号,再将item返回即可
        :param item: 
        :param spider: 
        :return: 
        """
        if item['text']:
            if len(item['text']) > self.limit:
                item['text'] = item['text'][0:self.limit].rstrip() + '...'
            return item
        else:
            return DropItem('Missing Text')


class MongoPipeline(object):
    def __init__(self, mongo_uri, mongo_db):
        """
        初始化
        """
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db
        
    @classmethod
    def from_crawler(cls, crawler):
        """
        类方法,用@classmethod标识,是一种依赖注入方法,它的参数就是crawler,通过crawler
        我们可以拿到全局配置的每一个配置信息,可以定义mongo_uri,mongo_db链接所需要的地址
        和数据库名称,拿到配置信息之后返回类对象即可,这方法主要就是获取settings.py的配置  
        :param crawler: 
        :return: 
        """
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DB')
        )
    
    def open_spider(self, spider):
        """
        启动爬虫,调用此方法
        :param spider: 
        :return: 
        """
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]
        
    def process_item(self, item, spider):
        """
        执行数据插入操作
        :param item: 
        :param spider: 
        :return: 
        """
        name = item.__class__.__name__
        self.db[name].insert(dict(item))
        return item
    
    def close_spider(self, spider):
        """
        当爬虫结束时调用
        :param spider: 
        :return: 
        """
        self.client.close()

配置settings.py

ITEM_PIPELINES = {
   'xxoo.pipelines.TextPipeline': 300,
    'xxoo.pipelines.MongoPipeline':400,
}
MONGO_URI = 'localhost'
MONGO_DB = 'xxoo'

赋值ITEM_PIPELINES字典,键名是pipeline的类名称, 键值是调用优先级,是一个数字,数字越小则对应pipeline越先被调用.

执行代码:

scrapy carwl min_spider

最后会在MOngoDB中创建一个叫做xxoo的数据库,QuoteItem的表