Scrapy框架

scrapy简介

Scrapy是一个用于抓取web站点和提取结构化数据的应用框架，它可用于广泛的有用应用，如数据挖掘、信息处理或历史存档。

可以参考scrapy的英文文档或者中文文档

整体的架构大致如下：

scrapy框架由scrapy引擎（scrapy engine）、调度器（scheduler）、下载器（downloader）、蜘蛛（spider）以及项目管道（item pipeline）组成。

工作流程大致如下：
首先scrapy引擎向调度器发送请求，调度器从url队列中取出一个url交给下载器，其次下载器向对应的服务器发送请求，得到响应后将下载网页内容，然后下载器把下载的网页内容交给蜘蛛进行解析，接着如果爬取到数据，则将数据交给项目管道进行加工处理，如果爬取到新的url，则保存在url队列中，进行新一轮的爬取。

五大组件及其中间件的功能如下：

• Scrapy引擎：Scrapy引擎相当于指令控制中心，控制整个系统的数据处理流程，触发事务处理流程，负责与各个模块进行通信；
• Scheduler（调度器）：维护待爬取的URL队列，当接受引擎发送的请求时，会从待爬取的URL队列中取出下一个URL返回给调度器。
• Downloader（下载器）：向对应的服务器发送下载页面的请求，用于下载网页内容，并把下载的网页内容交给蜘蛛处理。
• Spiders（蜘蛛）：制定要爬取的网站地址，选择所需数据内容，定义域名过滤规则和网页的解析规则等。
• Item Pipeline（项目管道）：处理由蜘蛛从网页中抽取的数据，主要任务是清洗、验证、过滤、去重和存储数据等。
• 中间件（Middlewares）：中间件是处于Scrapy引擎和Scheduler、Downloader、Spiders之间的构件，主要是处理它们之间的请求及响应。

scrapy框架爬取豆瓣电影top250
1、打开cmd，创建一个爬虫项目doubanmovie，会生成一些文件

scrapy startproject doubanmovie

scrapy.cfg
items.py
pipelines.py
middlewares.py
settings.py
spiders/
2、进入spider/文件夹，创建一个爬虫doubanspider，并指定需爬取的网页

scrapy genspider doubanspider movie.douban.com

3、items.py 定义爬取的数据内容

import scrapy


class DoubanmovieItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # 定义数据结构
    # 序号
    MovieNum = scrapy.Field()
    # 电影名
    MovieName = scrapy.Field()
    # 电影介绍
    Introduce = scrapy.Field()
    # 电影星级
    MovieStar = scrapy.Field()
    # 电影评论数
    MovieCount = scrapy.Field()
    # 电影描述
    Describe = scrapy.Field()

4、doubanspider.py 编写spider

import scrapy
from doubanmovie.items import DoubanmovieItem


class DoubanspiderSpider(scrapy.Spider):
    name = 'doubanspider'
    allowed_domains = ['movie.douban.com']
    start_urls = ['http://movie.douban.com/top250']

    def parse(self, response):
        
        MovieList = response.css("ol.grid_view li")
        for item in MovieList:

            MovieItem = DoubanmovieItem()
            MovieItem['MovieNum'] = item.css("div.pic em::text").get()
            MovieItem['MovieName'] = item.css("div.hd span.title::text").get()
            try:
                MovieItem['Introduce'] = item.css("div.bd p::text").get().strip().replace(" ", "")
            except Exception as e:
                MovieItem['Introduce'] = ''
            MovieItem['MovieStar'] = item.css("div.star span.rating_num::text").get()
            MovieItem['MovieCount'] = item.css("div.star span::text").get()
            try:
                MovieItem['Describe'] = item.css("p.quote::text").get().strip().replace(" ", "")
            except Exception as e:
                MovieItem['Describe'] = ''
            yield MovieItem

        # 爬取下一页
        NextLink = response.css("div.paginator span.next a::text").getall()
        if "后页>" in NextLink:
            NextPage = response.css("div.paginator span.next a::attr(href)").getall()
            print(NextPage)
            yield scrapy.Request(self.start_urls[0] + NextPage[len(NextPage)-1], callback=self.parse)

5、pipelines.py 保存数据

import json


class DoubanmoviePipeline:
    def __init__(self):
        self.f = open("douban.json", "w", encoding='utf-8')

    def process_item(self, item, spider):
        content = json.dumps(dict(item), ensure_ascii=False) + ",\n"
        self.f.write(content)
        return item
        
    def close_spider(self, spider):
        self.f.close()

6、settings.py 设置文件

BOT_NAME = 'doubanmovie'

SPIDER_MODULES = ['doubanmovie.spiders']
NEWSPIDER_MODULE = 'doubanmovie.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
# USER_AGENT = 'doubanmovie (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
# CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
# CONCURRENT_REQUESTS_PER_DOMAIN = 16
# CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
# COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
# TELNETCONSOLE_ENABLED = False

# Override the default request headers:
# DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
# }

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# SPIDER_MIDDLEWARES = {
#    'doubanmovie.middlewares.DoubanmovieSpiderMiddleware': 543,
# }

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# DOWNLOADER_MIDDLEWARES = {
#    'doubanmovie.middlewares.DoubanmovieDownloaderMiddleware': 543,
# }
DOWNLOADER_MIDDLEWARES = {
    'doubanmovie.middlewares.MyUserAgent': 544,
}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
# EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
# }

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    'doubanmovie.pipelines.DoubanmoviePipeline': 300,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
# AUTOTHROTTLE_ENABLED = True
# The initial download delay
# AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
# AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
# AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
# AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
# HTTPCACHE_ENABLED = True
# HTTPCACHE_EXPIRATION_SECS = 0
# HTTPCACHE_DIR = 'httpcache'
# HTTPCACHE_IGNORE_HTTP_CODES = []
# HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

7、middlewares.py 编写中间件

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals
import random
# useful for handling different item types with a single interface
# from itemadapter import is_item, ItemAdapter


class DoubanmovieSpiderMiddleware:
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the spider middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_spider_input(self, response, spider):
        # Called for each response that goes through the spider
        # middleware and into the spider.

        # Should return None or raise an exception.
        return None

    def process_spider_output(self, response, result, spider):
        # Called with the results returned from the Spider, after
        # it has processed the response.

        # Must return an iterable of Request, or item objects.
        for i in result:
            yield i

    def process_spider_exception(self, response, exception, spider):
        # Called when a spider or process_spider_input() method
        # (from other spider middleware) raises an exception.

        # Should return either None or an iterable of Request or item objects.
        pass

    def process_start_requests(self, start_requests, spider):
        # Called with the start requests of the spider, and works
        # similarly to the process_spider_output() method, except
        # that it doesn’t have a response associated.

        # Must return only requests (not items).
        for r in start_requests:
            yield r

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)


class DoubanmovieDownloaderMiddleware:
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        return None

    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        return response

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

class MyUserAgent(object):
    def process_request(self, request, spider):
        USER_AGENT_list = [
            'MSIE (MSIE 6.0; X11; Linux; i686) Opera 7.23',
            'Opera/9.20 (Macintosh; Intel Mac OS X; U; en)',
            'Opera/9.0 (Macintosh; PPC Mac OS X; U; en)',
            'iTunes/9.0.3 (Macintosh; U; Intel Mac OS X 10_6_2; en-ca)',
            'Mozilla/4.76 [en_jp] (X11; U; SunOS 5.8 sun4u)',
            'iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2)',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20120813 Firefox/16.0',
            'Mozilla/4.77 [en] (X11; I; IRIX;64 6.5 IP30)',
            'Mozilla/4.8 [en] (X11; U; SunOS; 5.7 sun4u)'
        ]
        UserAgent = random.choice(USER_AGENT_list)
        request.headers['User-Agent'] = UserAgent

8、运行爬虫，打开cmd进入spiders文件，输入命令

scrapy crawl doubanspider

9、爬取结果

参考：
1、https://blog.51cto.com/13389043/2348849
2、https://www.jianshu.com/p/a6d3db78bbc9