爬虫笔记（五）--Scrapy

scrapy

用户需要写的只有spider模块和item pipelines模块
downloader模块，scheduler模块，engine模块已有实现
只经过简单的配置就可实现框架功能，但不用于测试，先用request再移植到scrapy中进行爬取

1. engine：控制所有模块间的数据流，根据条件触发事件
2. downloader：根据请求下载网页
3. scheduler：对所有的爬取请求进行调度管理，决定谁先爬谁后爬，爬过的url如何处理，去重，过滤
4. downloader middleware：实施engine，scheduler和downloader之间用户可配置的控制，用来修改，丢弃，新增请求或响应，用户可修改配置代码
5. spider：解析 downloader返回的响应，产生爬取项（scraped item），产生额外的爬取请求
6. item pipelines：以流水线方式处理spider产生的爬取项，可能操作包括清理，检验查重爬取项中的html数据，将数据存储到数据库
7. spider middlewaare：对请求和爬取项的再处理，修改，丢弃新增请求或爬取项

requests VS. Scrapy

相同点：1.两者都可以进行页面请求和爬取，python爬虫的两个重要技术路线
	    2.两者可用性都号，文档丰富，入门简单
	    3.两者都没有处理js，提交表单，应对验证码等功能（可扩展）

requests	Scrapy
页面级爬虫	网站级爬虫
功能库	框架
并发性考虑不足	并发性好
重点在于页面下载	重点在于爬虫结构
定制灵活	一般定制灵活，深度定制困难

request发请求等待响应的时候无法处理其他事
爬取速度快慢要根据具体情况而定

Request和Response类

Request

Request对象表示一个HTTP请求
由Spider生成，由downloader执行

class scrapy.http.Request()

属性或方法	说明
url	Request对应的请求url地址
callback	指定本次请求响应的回调函数
method	对应的请求方法，‘GET’‘POST’
proxy	代理
headers	字典类型的请求头
cookies	cookie,添加时需要键值对形式(" “:” ")
encoding	编码方式
priority	默认为0
dont_filter	是否过滤请求（已发送），默认True
errback	请求失败的回调函数，可被try，except代替
body	请求内容主体，字符串类型
meta	用户添加的扩展信息，字典类型，在Scrapy内部模块间传递信息使用
.copy()	复制该请求
.FormRequest()	发送POST请求

Response

Response对象表示一个HTTP响应
由downloader生成，由spider处理

class scrapy.http.Response()

属性或方法	说明
url	Response对应的请求url地址
status	HTTP状态码，默认是200
text	文本数据
meta	响应的meta，从中可以拿到响应的cookies
headers	response对应的头部信息
body	response对应的内容信息，二进制流数据，用于存储媒体流文件
flags	一组标记
request	产生Response类型对应的Request对象
.copy()	复制该响应
.xpath()	用来做xpath解析，还需要用.getall()或者.extract()获取内容

scrapy源码

spider

ctrl+鼠标单击查看函数信息

class Spider(object_ref):
    """Base class for scrapy spiders. All spiders must inherit from this
    class.
    """
       #所有spider的基类 

    name = None #类属性，爬虫名
    custom_settings = None

    def __init__(self, name=None, **kwargs):
        if name is not None:
            self.name = name#如果爬虫名非空，按照输入的赋值
        elif not getattr(self, 'name', None):#如果没有名字则抛出异常
            raise ValueError("%s must have a name" % type(self).__name__)
        self.__dict__.update(kwargs)
        if not hasattr(self, 'start_urls'):#如果没有start_urls则设置为空列表
            self.start_urls = []
        @property

#日志记录
    def logger(self):
        logger = logging.getLogger(self.name)
        return logging.LoggerAdapter(logger, {'spider': self})
    def log(self, message, level=logging.DEBUG, **kw):
        """Log the given message at the given log level

        This helper wraps a log call to the logger within the spider, but you
        can use it directly (e.g. Spider.logger.info('msg')) or use any other
        Python logger too.
        """
        self.logger.log(level, message, **kw)
        
#创建crawler
    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = cls(*args, **kwargs)
        spider._set_crawler(crawler)
        return spider
        
#设置crawler，抛出信号
    def _set_crawler(self, crawler):
        self.crawler = crawler
        self.settings = crawler.settings
        crawler.signals.connect(self.close, signals.spider_closed)    

#起始请求方法
    def start_requests(self):
        cls = self.__class__
        if method_is_overridden(cls, Spider, 'make_requests_from_url'):
            warnings.warn(
                "Spider.make_requests_from_url method is deprecated; it "
                "won't be called in future Scrapy releases. Please "
                "override Spider.start_requests method instead (see %s.%s)." % (
                    cls.__module__, cls.__name__
                ),
            )
            for url in self.start_urls:#从start_urls中取出url，start_urls中可写入多个url
                yield self.make_requests_from_url(url)
        else:
            for url in self.start_urls:
                yield Request(url, dont_filter=True)
            
    
    def make_requests_from_url(self, url):#根据url创建request对象
        """ This method is deprecated. """
        return Request(url, dont_filter=True)#默认发请求不过滤，没有cookie，headers等参数，需要携带时需要重载

    
    def parse(self, response):
        raise NotImplementedError('{}.parse callback is not defined'.format(self.__class__.__name__))#必须重写parse方法，否则抛出异常

    @classmethod
    def update_settings(cls, settings):
        settings.setdict(cls.custom_settings or {}, priority='spider')

    @classmethod
    def handles_request(cls, request):
        return url_is_from_spider(request.url, cls)


    @staticmethod
    def close(spider, reason):
        closed = getattr(spider, 'closed', None)
        if callable(closed):#如果属性中有close那么就关闭爬虫
            return closed(reason)

---

Item

Item对象表示一个从HTML页面中提取的信息内容
由spider生成，由Item pipeline处理
Item类似字典类型，可以按照字典类型操作

---

pipelines

方法名不能改，参数不要改，函数从spider中接收item并存储

---

settings

存放spider的位置

---

全局的user-agent，除非被headers中的参数覆盖，电脑和浏览器不变全局user-agent也不变，当访问网站次数过多时可能需要更换

---

启用pipelines，参数为优先级，数字越小优先级越高

---

是否遵守robots协议

---

全局请求头

---

最大并发连接数，linux为1024，windows为512

---

配置爬虫中间件，键为中间件所在位置，值为优先级

---

配置下载器中间件，键为中间件所在位置，值为优先级

---

配置爬取深度，不配置

---

不启用cookies，不配置

---

不使用telnet链接即拨号上网，不配置

---

是否允许拓展，不配置

---

限速操作，不配置

---

设置缓存，一般不配置

---

日志配置

在setting中设置

是否显示日志

---

日志级别（向下兼容）
1. DEBUG : 程序基本调试信息
2. INFO ： 基本提示信息
3. WARNING ： 警告
4. ERROR ： 一般错误
5. CRITICAL ： 严重错误，可能引起程序崩溃

---

主动抛出日志

---

断点续传

原理：调度器的内存队列和磁盘队列
内存队列：在内存中保存request对象的容器
	   特点：快，无法持久化
磁盘队列：在磁盘中保存request对象的容器
	  特点：慢，持久化
scrapy默认使用内存队列，如果配置jobdir可以把调度器中的数据进行持久化

JOBDIR=" "#设置持久化的目录
request如果边爬取边存数据则也算持久化（使用update）

middleware

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals


class LearnSpiderMiddleware(object):
#spider中间件不是所有方法都需要被定义，如果没有定义则pass


#创建爬虫时调用
    @classmethod
    def from_crawler(cls, crawler):
        
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)#信号提示爬虫启动
        return s

#当每一个响应穿过爬虫中间件进入spider时被调用    
    def process_spider_input(self, response, spider):

        # 返回none或者抛出异常
        return None

#处理完response之后调用
    def process_spider_output(self, response, result, spider):
        # 必须返回一个可迭代的request对象或者item对象或者字典
        for i in result:
            yield i

#process_spider_input抛出异常时调用
    def process_spider_exception(self, response, exception, spider):
        # 返回none或者request或者item或者字典
        # 不需要注意，之后为pass
        pass

#start_requests被调用时
    def process_start_requests(self, start_requests, spider):
        #类似于process_spider_output() method, 
        #只能返回request
        for r in start_requests:
            yield r

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

#下载器中间件，可以在发送给downloader之前添加数据
class LearnDownloaderMiddleware(object):
 	#spider中间件不是所有方法都需要被定义，如果没有定义则pass

#创建时调用    
    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

#当request穿过下载器中间件（未到达）时调用
    def process_request(self, request, spider):
        # Must either:
        # 返回None: 继续
        # 返回Response给spider
        # 返回Request给调度器
        # 抛出异常调用process_exception()   
        return None

#当Response从下载器返回时调用
    def process_response(self, request, response, spider):
        # Must either;
        # 返回Response给spider
        # 返回Request给调度器
        # 抛出异常调用process_exception()
        return response

#异常处理
    def process_exception(self, request, exception, spider):       
        # 返回None: 跳过处理异常
        # 返回Response：停止处理异常的动作链
        # 返回Request：停止处理异常的动作链
        return response
        pass

#提示爬虫启动
    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

scheduler

class Scheduler(object):
    """
    调度器可让多个requests入队，并且让下一个request去下载，并具有去重机制

    优先级和队列不会被调度器执行，需要用户去设置
    
    (defined by :setting:`SCHEDULER_PRIORITY_QUEUE`) uses these priorities
    to dequeue requests in a desired order.

    调度器使用两个优先级队列实例，默认使用磁盘队列,当磁盘队列不能处理时使用内存队列

    :setting:`SCHEDULER_MEMORY_QUEUE` and
    :setting:`SCHEDULER_DISK_QUEUE` allow to specify lower-level queue classes
    which PriorityQueue instances would be instantiated with, to keep requests
    on disk and in memory respectively.

    总之，调度器是一个池由优先级队列和应用反馈逻辑的调度器，并且可以控制去重策略
    """
    def __init__(self, dupefilter, jobdir=None, dqclass=None, mqclass=None,
                 logunser=False, stats=None, pqclass=None, crawler=None):
        self.df = dupefilter#去重策略
        self.dqdir = self._dqdir(jobdir)#设置在磁盘队列中的工作路径
        self.pqclass = pqclass
        self.dqclass = dqclass
        self.mqclass = mqclass
        self.logunser = logunser
        self.stats = stats
        self.crawler = crawler

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings#导入配置
        dupefilter_cls = load_object(settings['DUPEFILTER_CLASS'])#setting中去重策略的类
        dupefilter = create_instance(dupefilter_cls, settings, crawler)#创建去重策略
        pqclass = load_object(settings['SCHEDULER_PRIORITY_QUEUE'])#加载优先级队列
        if pqclass is PriorityQueue:#优先级在优先级队列中抛出警告
            warnings.warn("SCHEDULER_PRIORITY_QUEUE='queuelib.PriorityQueue'"
                          " is no longer supported because of API changes; "
                          "please use 'scrapy.pqueues.ScrapyPriorityQueue'",
                          ScrapyDeprecationWarning)
            from scrapy.pqueues import ScrapyPriorityQueue
            pqclass = ScrapyPriorityQueue

        dqclass = load_object(settings['SCHEDULER_DISK_QUEUE'])#加载磁盘队列
        mqclass = load_object(settings['SCHEDULER_MEMORY_QUEUE'])#加载内存队列
        logunser = settings.getbool('LOG_UNSERIALIZABLE_REQUESTS',
                                    settings.getbool('SCHEDULER_DEBUG'))#记录日志
        return cls(dupefilter, jobdir=job_dir(settings), logunser=logunser,
                   stats=crawler.stats, pqclass=pqclass, dqclass=dqclass,
                   mqclass=mqclass, crawler=crawler)#生成包装好的对象

    def has_pending_requests(self):
        return len(self) > 0

    def open(self, spider):#打开爬虫
        self.spider = spider
        self.mqs = self._mq()
        self.dqs = self._dq() if self.dqdir else None
        return self.df.open()

    def close(self, reason):#关闭爬虫
        if self.dqs:#如果磁盘队列中没有值则关闭
            state = self.dqs.close()
            self._write_dqs_state(self.dqdir, state)
        return self.df.close(reason)

    def enqueue_request(self, request):
        if not request.dont_filter and self.df.request_seen(request):#如果requests已重复并在已爬取的队列中，就抛出日志，并return false
            self.df.log(request, self.spider)
            return False
        dqok = self._dqpush(request)#不重复则push进dq
        if dqok:#如果磁盘队列非空就应用磁盘队列
            self.stats.inc_value('scheduler/enqueued/disk', spider=self.spider)
        else:#否则request push进内存队列
            self._mqpush(request)
            self.stats.inc_value('scheduler/enqueued/memory', spider=self.spider)
        self.stats.inc_value('scheduler/enqueued', spider=self.spider)
        return True

    def next_request(self):
        request = self.mqs.pop()#从内存队列中取出request交给spider
        if request:#有值
            self.stats.inc_value('scheduler/dequeued/memory', spider=self.spider)
        else:#无值则从磁盘队列中取
            request = self._dqpop()
            if request:
                self.stats.inc_value('scheduler/dequeued/disk', spider=self.spider)
        if request:#不管是dq还是mq拿出了request则加入到已采集的队列中防止爬到重复对象
            self.stats.inc_value('scheduler/dequeued', spider=self.spider)
        return request

    def __len__(self):
        return len(self.dqs) + len(self.mqs) if self.dqs else len(self.mqs)

    def _dqpush(self, request):
        if self.dqs is None:
            return
        try:
            self.dqs.push(request, -request.priority)
        except ValueError as e:  # non serializable request
            if self.logunser:
                msg = ("Unable to serialize request: %(request)s - reason:"
                       " %(reason)s - no more unserializable requests will be"
                       " logged (stats being collected)")
                logger.warning(msg, {'request': request, 'reason': e},
                               exc_info=True, extra={'spider': self.spider})
                self.logunser = False
            self.stats.inc_value('scheduler/unserializable',
                                 spider=self.spider)
            return
        else:
            return True

    def _mqpush(self, request):
        self.mqs.push(request, -request.priority)

    def _dqpop(self):
        if self.dqs:
            return self.dqs.pop()

    def _newmq(self, priority):
        """ Factory for creating memory queues. """
        return self.mqclass()

    def _newdq(self, priority):
        """ Factory for creating disk queues. """
        path = join(self.dqdir, 'p%s' % (priority, ))
        return self.dqclass(path)

    def _mq(self):
        """ Create a new priority queue instance, with in-memory storage """
        return create_instance(self.pqclass, None, self.crawler, self._newmq,
                               serialize=False)

    def _dq(self):
        """ Create a new priority queue instance, with disk storage """
        state = self._read_dqs_state(self.dqdir)
        q = create_instance(self.pqclass,
                            None,
                            self.crawler,
                            self._newdq,
                            state,
                            serialize=True)
        if q:
            logger.info("Resuming crawl (%(queuesize)d requests scheduled)",
                        {'queuesize': len(q)}, extra={'spider': self.spider})
        return q

    def _dqdir(self, jobdir):
        """ Return a folder name to keep disk queue state at """
        if jobdir:
            dqdir = join(jobdir, 'requests.queue')
            if not exists(dqdir):
                os.makedirs(dqdir)
            return dqdir

    def _read_dqs_state(self, dqdir):
        path = join(dqdir, 'active.json')
        if not exists(path):
            return ()
        with open(path) as f:
            return json.load(f)

    def _write_dqs_state(self, dqdir, state):
        with open(join(dqdir, 'active.json'), 'w') as f:
            json.dump(state, f)

常用命令

scrapy是为持续运行设计的专业爬虫框架，提供操作的scrapy命令行，在scrapy框架下一个工程是一个最大单元相当于一个大的scrapy框架，在scrapy框架中可以有多个爬虫，每一个爬虫相当于一个scrapy模块
>scrapy<command>[options][args]

命令	说明	格式
startproject	创建一个新工程	scrapy startproject <name>[dir]
genspider	创建一个爬虫	scrapy genspider [options]<name><domain>
settings	获取爬虫配置信息	scrapy settings [options]
crawl	运行一个爬虫	scrapy crawl <spider>
list	运行工程中所有爬虫	scrapy list
shell	启动url调试命令行	scrapy shell [url]

为什么使用命令行

11. 命令行更容易自动化，适合脚本控制
12. 给程序员使用，功能更重要
    
    

生成爬虫

1.建立一个scrapy爬虫工程

2.在工程中产生一个scrapy爬虫


会在工程目录里的spiders中生成新的py文件

demo.py

# -*- coding: utf-8 -*-
import scrapy
class DemoSpider(scrapy.Spider):#必须继承于scrapy.spider
    name = 'demo'
    #命令中赋值name
    allowed_domains = ['python123.io']
    #命令中提交的域名，爬虫在爬取网站时只能爬这个网站以下的相关链接
    start_urls = ['http://python123.io/']
    #爬取的初始页面
    def parse(self, response):
        #用于处理响应，解析内容形成字典，发现新的url爬取请求
        pass

3.配置爬虫

class DemoSpider(scrapy.Spider):
    name = 'demo'
    start_urls = ['https://python123.io/ws/demo.html']
    def parse(self, response):
        #response为返回内容对应的对象
        fname = response.url.split('/')[-1]
        with open(fname,'wb') as f:
            f.write(reponse.body)
        self.log('Saved file %s.' % name)
        pass

4.运行爬虫

yield关键字

	主要与循环在一起使用
	yield= 生成器
	生成器是一个不断产生值的函数
	包含yield语句的函数是一个生成器
	生成器每次产生一个值，函数被冻结，被唤醒后再产生一个值
	下次执行时从被冻结处开始执行

为什么要有生成器

• 节省存储空间
• 响应更迅速
• 使用更灵活

UA，cookies，headers添加

1. spider中生成Request对象的时候传入headers
2. 在settings中配置全局UA
3. 在settings中配置全局headers

中间件添加

股票实例

步骤

1.建立工程和Spider模块
命令行中输入命令：

scrapy startproject Stocks
cd Stocks
scrapy genspider demo quote.cfi.cn/stockList.aspx#http：//之后的东西

修改spiders/stocks.py文件
2.编写Spider

• 配置demo.py文件
• 修改对返回页面的处理
• 修改对新增url爬取请求的处理

3.编写pipelines

• 配置pipelines.py文件
• 定义对爬取项（Scraped Item）的处理类
• 配置ITEM_PIPELINES选项settings.py文件中找到参数ITEM_PIPELINES并修改为自己写的

代码

spider

# -*- coding: utf-8 -*-
import scrapy
import re
class DemoSpider(scrapy.Spider):
    name = 'demo'
    start_urls = ['http://quote.cfi.cn/stockList.aspx/']

    def parse(self, response):
        for href in response.css('a::attr(href)').extract():
            try:
                stock = re.findall(r'\d{6}',href)[0]
                url = 'http://quote.cfi.cn/' + stock + '.html'
                yield scrapy.Request(url, callback = self.parse_stock)
            except:
                continue
        pass
    def parse_stock(self, response):
        infoDict = {}
        stockInfo = response.css('#act_quote')[0]
        #id查找使用#
        nmae1 = stockInfo.css('.Lfont')[0]
        name = nmae1.css('::text').extract()
        Item = response.css('#quotetab_stock')[0]
        item = Item.css('td')
        count = 0
        for Info in item:
            if(count==10):
                break
            info ="".join(Info.css('::text').extract())
            # print(info)
            key = info.split(':')[0]
            # print(key)
            val = info.split(':')[1]
            infoDict[key] = val
            count=count+1
        infoDict.update({'股票名称':name[0].split()[0]})
        yield  infoDict
        #class查找使用.

pipelines

class StocksPipeline(object):
    #对每一个item进行处理的过程
    def process_item(self, item, spider):
        return item
class StocksInfoPipeline(object):
    def open_spider(self,spider):
        #爬虫被调用时启动的方法
        self.f = open('股票.txt','w')
    def close_spider(self,spider):
        self.f.close()
    def process_item(self, item, spider):
        try :
            line = str(dict(item))+'\n'
            self.f.write(line)
        except:
            pass
        return item

招聘网站实例

spider

# -*- coding: utf-8 -*-
from ..items import LearnItem
import scrapy
class Job51Spider(scrapy.Spider):
    name = 'job51' #
   # allowed_domains = ['51job.com']
   #  start_urls = ['https://search.51job.com/list/211000,000000,0000,00,9,99,%25E9%2594%2580%25E5%2594%25AE,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=']
    def start_requests(self):#分页，添加cookie，headers等情况不适用start_url,需要重载start_requests
        for i in range(50):
            url = "http://search.51job.com/list/020000,000000,0000,00,9,99,%2B,2," + str(i) + ".html?lang=c&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare="
            yield scrapy.Request(url=url)
            #for循环中使用return则循环结束
    def parse(self, response):
        job_name = response.xpath("//div[@class='el']/p/span/a/@title").getall()
        #selector对象getall()方法，返回列表
        c_name = response.xpath("//div[@class='el']/span/a/text()").getall()
        item = LearnItem()
        #实例化对象，此时item为字典
        for name in job_name:
            index = job_name.index(name)
            #获取索引值
            new_cname = c_name[index]
            #公司名称
            item['j_name']=name
            #往item对象存工作名
            item['c_name']=new_cname
            yield item

items

import scrapy
class LearnItem(scrapy.Item):
    # define the fields for your item here like:
    j_name = scrapy.Field()
    #工作名
    c_name = scrapy.Field()
    #公司名
    pass

pipelines

class LearnPipeline(object):
    def process_item(self, item, spider):
        print(item)
        return item

小说实例

scrapy异步框架处理小说内容会乱序

spider

import scrapy
from ..items import BiqugeItem
class BiqugeSpider(scrapy.Spider):
    name = "biquge"
    def start_requests(self):
        url = "http://www.xbiquge.la/xiaoshuodaquan/"
        yield scrapy.Request(url,callback=self.parse)
        #设置回调函数为自身
    def parse(self, response):
        book_urls = response.xpath("//div[@class='novellist']//a/@href").getall()
        #获取列表页所有小说的url
        for book_url in book_urls:
            yield scrapy.Request(url=book_url,callback=self.parse1)
            #start
    def parse1(self, response):
        chapter_urls = response.xpath("//div[@id='list']/dl//dd/a/@href").getall()
        #获取章节url
        new_cha_urls = ["http://www.xbiquge.la"+i for i in chapter_urls]
        #拼接具体章节url
        for cha_url in new_cha_urls:
            yield scrapy.Request(url=cha_url,callback=self.parse2)
    def parse2(self, response):
        item = BiqugeItem()
        cont = response.xpath("//div[@id='content']/text()").getall()
        cha_name = response.xpath("//h1/text()").getall()[0]
        #获取章节名字
        s = ""
        #空字符串来存放小说内容
        for i in cont:
            s += i.strip()
            #去空格
        item['content']=s
        item['cha_name']=cha_name
        yield item

items

class BiqugeItem(scrapy.Item):
    # define the fields for your item here like:
    content = scrapy.Field()
    #内容
    cha_name = scrapy.Field()
	#章节名

pipelines

class BiqugePipeline(object):
    def process_item(self, item, spider):
        print(item)
        return item