爬虫：Scrapy爬虫框架

Scrapy简介

通用爬虫框架流程

Scrapy 框架流程

Scrapy组件

Scrapy主要包括了以下组件：

• 引擎(Scrapy)
  用来处理整个系统的数据流处理, 触发事务(框架核心)
• 调度器(Scheduler)
  用来接受引擎发过来的请求, 压入队列中, 并在引擎再次请求的时候返回. 可以想像成一个URL（抓取网页的网址或者说是链接）的优先队列, 由它来决定下一个要抓取的网址是什么, 同时去除重复的网址
• 下载器(Downloader)
  用于下载网页内容, 并将网页内容返回给蜘蛛(Scrapy下载器是建立在twisted这个高效的异步模型上的)
• 爬虫(Spiders)
  爬虫是主要干活的, 用于从特定的网页中提取自己需要的信息, 即所谓的实体(Item)。用户也可以从中提取出链接,让Scrapy继续抓取下一个页面
• 项目管道(Pipeline)
  负责处理爬虫从网页中抽取的实体，主要的功能是持久化实体、验证实体的有效性、清除不需要的信息。当页面被爬虫解析后，将被发送到项目管道，并经过几个特定的次序处理数据。
• 下载器中间件(Downloader Middlewares)
  位于Scrapy引擎和下载器之间的框架，主要是处理Scrapy引擎与下载器之间的请求及响应。
• 爬虫中间件(Spider Middlewares)
  介于Scrapy引擎和爬虫之间的框架，主要工作是处理蜘蛛的响应输入和请求输出。
• 调度中间件(Scheduler Middewares)
  介于Scrapy引擎和调度之间的中间件，从Scrapy引擎发送到调度的请求和响应。

Scrapy运行流程

1. 引擎从调度器中取出一个链接(URL)用于接下来的抓取
2. 引擎把URL封装成一个请求(Request)传给下载器
3. 下载器把资源下载下来，并封装成应答包(Response)
4. 爬虫解析Response
5. 解析出实体（Item）,则交给实体管道进行进一步的处理
6. 解析出的是链接（URL）,则把URL交给调度器等待抓取

Scrapy的安装

Linux下的安装(包括mac)

pip install scrapy

Windows下的安装

1. 下载twisted 
	http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
	
2. 安装wheel 
	pip3 install wheel

3. 安装twisted 
	进入下载目录，执行 pip3 install Twisted‑18.7.0‑cp36‑cp36m‑win_amd64.whl

4. 安装pywin32
	pip3 install pywin32

5. 安装scrapy 
	pip3 install scrapy 

基本命令

1. scrapy startproject 项目名称
	在当前目录中创建一个项目文件
	
2. scrapy genspider [-t template] <name> <domain>
	创建爬虫应用
		如：
     	 	scrapy gensipider -t basic oldboy oldboy.com
      		scrapy gensipider -t xmlfeed autohome autohome.com.cn
		或者简单直接: 
			 scrapy gensipider app名  要爬取的域名
	PS:
 		查看所有命令：scrapy gensipider -l
		查看模板命令：scrapy gensipider -d 模板名称
		
3. scrapy list
	展示爬虫应用列表
	
4. scrapy crawl 爬虫应用名称
	运行单独爬虫应用
备注：
	scrapy crawl 应用名称  表示以日志的形式运行爬虫应用，可以在后面加 --nolog  取消日志
    scrapy crawl 名称  --nolog

项目文件说明

• scrapy.cfg 项目的主配置信息。（真正爬虫相关的配置信息在settings.py文件中）
• items.py 设置数据存储模板，用于结构化数据，如：Django的Model
• pipelines 数据处理行为，如：一般结构化的数据持久化
• settings.py 配置文件，如：递归的层数、并发数，延迟下载等
• spiders 爬虫目录，如：创建文件，编写爬虫规则

项目案例

项目介绍

为了充分利用网上大数据资源,让用户能够方便利用影视信息,采用基于 Scrapy 框架的爬虫技术,开发了检索电影信息的搜索引擎。对豆瓣网站的影视信息进行爬取,以方便用户准确获取最新的电影信息。

项目代码

以“豆瓣电影”为爬取目 标,爬取网站中的影视信息。主要包括网站排名 “ Top250 ”和喜剧、动作类电影的电影名称、电影评分、电影导演, 电影上映时间以及电影评语。

创建工程

scrapy startproject DouBan

创建爬虫程序

cd DouBan/
scrapy genspider douban 'douban.com'

自动创建目录及文件

编写爬虫文件（douban.py）

# -*- coding: utf-8 -*-
import scrapy
from scrapy import Request
from DouBan.items import DoubanItem
import copy


class DoubanSpider(scrapy.Spider):
    name = 'douban'
    allowed_domains = ['douban.com']
    # start_urls = ['http://douban.com/']
    start_urls = ['https://movie.douban.com/top250']
    url = 'https://movie.douban.com/top250'

    def parse(self, response):
        items = DoubanItem()
        # with open('douban.html', 'w') as f:
        #     f.write(response.text)
        movies = response.xpath("//ol[@class='grid_view']/li")
        for movie in movies:
            title = movie.xpath(".//span[@class='title']/text()").extract()[0]
            rating_num = movie.xpath(".//span[@class='rating_num']/text()").extract()[0]
            # <span class="inq">希望让人自由。</span>
            inq = movie.xpath(".//span[@class='inq']/text()").extract()
            if inq:
                inq = inq[0]
            items['inq'] = inq
            items['rating_num'] = rating_num
            items['title'] = title

            # 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p1454261925.jpg'
            items['image_url'] = movie.xpath('.//div[@class="pic"]/a/img/@src').extract()[0]
            # print("image url: ", item['image_url'])

            items['detail_url'] = movie.xpath('.//div[@class="hd"]//a/@href').extract()[0]
            # print("detail url: ", item['detail_url'])
            # yield items
            yield Request(items['detail_url'], meta={'item': copy.deepcopy(items)}, callback=self.detailParse)

        # """
        #    <span class="next">
        #    <link rel="next" href="?start=50&amp;filter=">
        #    <a href="?start=50&amp;filter=">??&gt;</a>
        #    </span>
        #    """
        # nextLink = response.xpath('.//span[@class="next"]/link/@href').extract()  # 返回列表
        # if nextLink:
        #     nextLink = nextLink[0]
        #     print('Next Link: ', nextLink)
        #     yield Request(self.url + nextLink, callback=self.parse)

    def detailParse(self, response):
        items = response.meta['item']
        # print(items, '111111111111')
        items['movieLength'] = response.xpath(".//span[@property='v:runtime']/text()").extract()[0]
        print(items, '333333333333333')
        yield copy.deepcopy(items)

编辑item文件

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class DoubanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    inq = scrapy.Field()
    rating_num = scrapy.Field()
    title = scrapy.Field()
    image_url = scrapy.Field()
    detail_url = scrapy.Field()
    image_path = scrapy.Field()
    movieLength = scrapy.Field()

编辑pipelines文件

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import json

import pymysql
import scrapy
from scrapy.exceptions import DropItem
from scrapy.pipelines.images import ImagesPipeline


class DoubanPipeline(object):
    def process_item(self, item, spider):
        return item


class AddScoreNum(object):
    def process_item(self, item, spider):
        if item['rating_num']:
            rating_num = float(item['rating_num'])
            item['rating_num'] = str(rating_num + 1)
            return item
        else:
            raise Exception('没有爬取到rating_num')


class JsonWriterPipeline(object):
    def open_spider(self, spider):
        self.file = open('douban.json', 'w')

    def process_item(self, item, spider):
        line = json.dumps(dict(item), indent=4, ensure_ascii=False)
        self.file.write(line)
        return item

    def close_spider(self, spider):
        self.file.close()


class MysqlPipeline(object):
    def open_spider(self, spider):
        self.connect = pymysql.connect(
            host='127.0.0.1',
            port=3306,
            db='scrapyProject',
            user='root',
            passwd='westos',
            charset='utf8',
            use_unicode=True
        )

        self.cursor = self.connect.cursor()
        a = "create table if not exists douBanTop250(title varchar(50) unique,rating_num float,inq varchar(100));"
        print(a)
        self.cursor.execute(a)

    def process_item(self, item, spider):
        insert_sqli = "insert into douBanTop250(title, rating_num, inq) " \
                      "values('%s', '%s', '%s');" \
                      % (item['title'], item['rating_num'], item['inq'])
        print(insert_sqli)
        try:
            self.cursor.execute(insert_sqli)
            print(111)
        except Exception as e:
            self.connect.rollback()
            print(222)
        else:
            self.connect.commit()
            print(333)

        return item

    def close_spider(self, spider):
        self.connect.commit()
        self.cursor.close()
        self.connect.close()


class MyImagesPipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
        yield scrapy.Request(item['image_url'])

    def item_completed(self, results, item, info):
        """
        :param results:
            [(True,  {'url': 'https://img3.doubanio.com/view/photo/s_ratio_poster/public/p1454261925.jpg',
                'path': 'full/e9cc62a6d6a0165314b832b1f31a74ca2487547a.jpg',
                'checksum': '5d77f59d4d634b795780b2138c1bf572'})]
        :param item:
        :param info:
        :return:
        """
        # for result in results:
        #     print("result: ", result)
        
        # isok = True/False
        image_paths = [x['path'] for isok, x in results if isok]
        # print("image_paths: ", image_paths[0])
        if not image_paths:
            raise DropItem("Item contains no images")
        item['image_path'] = image_paths[0]
        return item

编辑settings文件

# -*- coding: utf-8 -*-

# Scrapy settings for DouBan project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'DouBan'

SPIDER_MODULES = ['DouBan.spiders']
NEWSPIDER_MODULE = 'DouBan.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'DouBan (+http://www.yourdomain.com)'
from fake_useragent import UserAgent
ua = UserAgent()
USER_AGENT = ua.random

# Obey robots.txt rules
# ROBOTSTXT_OBEY = True
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'DouBan.middlewares.DoubanSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'DouBan.middlewares.DoubanDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'scrapy.pipelines.images.ImagesPipeline': 1,
   'scrapy.pipelines.files.FilesPipeline': 2,
   'DouBan.pipelines.MyImagesPipeline': 2,
   'DouBan.pipelines.AddScoreNum': 100,
   'DouBan.pipelines.JsonWriterPipeline': 200,
   'DouBan.pipelines.MysqlPipeline': 200
}

IMAGES_STORE = './images'
# FILES_STORE = './files'
IMAGES_EXPIRES = 30
# FILES_EXPIRES = 90
IMAGES_THUMBS = {
   'small': (50, 50),
   'big': (270, 270)
}
IMAGES_MIN_HEIGHT = 110
IMAGES_MIN_WIDTH = 110

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

项目效果

• 实现了页面解析爬取
• 爬取电影短评，分数，电影名，电影时长
• 对电影分数实现加一操作
• item数据保存为json文件
• 下载保存图片：电影缩略图，电影大图