第八十一篇 scrapy框架

心得：系统的学一下scrapy框架，其实爬虫没多难，最难的就是用不用心去分析网站，有没有一个整体的思路和框架。

一、Scrapy简介

scrapy在于爬取数据方面还是比较高效率的。Scrapy 使用了 Twisted(其主要对手是Tornado)异步网络框架来处理网络通讯，可以加快我们的下载速度，不用自己去实现异步框架，并且包含了各种中间件接口，可以灵活的完成各种需求。

windows安装scrapy

直接使用命令安装不成功可以下载whl格式的包安装，安装whl格式包需要安装wheel库

pip install wheel

因为scrapy框架基于Twisted,所以先要下载其whl包安装
http://www.lfd.uci.edu/~gohlke/pythonlibs/

pip install 对应版本的wheel包

安装pywin32

pip install pywin32

安装scrapy

pip install scrapy

linux安装scrapy

pip install scrapy

工程命令基本用法

创建第一个爬虫文件示例：

创建项目：

scrapy startproject first_demo

进入项目：

cd first_demo

创建爬虫文件：

第三个参数：爬虫文件名
第四个参数：爬虫的起始网站

scrapy genscrapy first_scrapy_file www.baidu.com

工程目录结构：

items.py:定义爬虫程序的数据模型

middlewares.py:定义数据模型中的中间件

pipelines.py:管道文件,负责对爬虫返回数据的处理

settings.py:爬虫程序设置,主要是一些优先级设置,优先级越高,值越小

scrapy.cfg:内容为scrapy的基础配置

二、爬取的简单应用

1 简单示例：爬取糗事百科（https://www.qiushibaike.com/text/）

分析网页抓取作者和内容并通过yield发送给管道存储
qiushibaike.py

# -*- coding: utf-8 -*-
import scrapy
from first_demo.items import FirstDemoItem  #导入items中定义的类，然后实例化


class FirstScrapyFileSpider(scrapy.Spider):
    name = 'first_scrapy_file'
    # allowed_domains = ['www.baidu.com']
    start_urls = ['https://www.qiushibaike.com/text/']

    def parse(self, response):
        # div_list = response.xpath('//div[@id="content-left"]/div')
        div_list = response.xpath("//div[@class='col1 old-style-col1']/div")
        for div in div_list:
            author = div.xpath('./div[1]/a[2]/h2/text()').extract_first()
            content = div.xpath('./a[1]/div[1]/span/text()').extract_first()

            # 实例化一个item类型的对象
            item = FirstDemoItem()
            # 将解析到的数据值存储到item对象中:why?
            item['author'] = author
            item['content'] = content
            # 将item对象提交给管道进行持久化存储
            yield item

定义两个字段
items.py

import scrapy

class FirstDemoItem(scrapy.Item):
    # define the fields for your item here like:
    author = scrapy.Field()
    content = scrapy.Field()

管道接收存储
pipelines.py

class FirstDemoPipeline:
    fp = None

    # 只会被执行一次(开始爬虫的时候执行一次)
    def open_spider(self, spider):
        print('开始爬虫!!!')
        self.fp = open('./job.txt', 'w', encoding='utf-8')

    # 爬虫文件没提交一次item,该方法会被调用一次
    def process_item(self, item, spider):
        self.fp.write(item['author'] + "\n" + item['content'] + '\n')
        return item

    def close_spider(self, spider):
        print('爬虫结束!!!')
        self.fp.close()
# 注意:默认情况下,管道机制并没有开启.需要手动在配置文件中进行开启

# 使用管道进行持久化存储的流程:
# 1.获取解析到的数据值
# 2.将解析的数据值存储到item对象(item类中进行相关属性的声明)
# 3.通过yild关键字将item提交到管道
# 4.管道文件中进行持久化存储代码的编写(process_item)
# 5.在配置文件中开启管道

settings.py

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#填写UA伪装
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

# Obey robots.txt rules #关闭robots验证
ROBOTSTXT_OBEY = False

#管道存储时，需要打开，300代表的是优先级，越小优先级越高
ITEM_PIPELINES = {
   'first_demo.pipelines.FirstDemoPipeline': 300,
}

开启爬取命令

不会打印log信息

scrapy crawl first_demo --nolog

2 抽屉爬取以及MongoDB,Redis,Mysql存储(https://dig.chouti.com/)

使用管道进行持久化存储的流程:
1.获取解析到的数据值
2.将解析的数据值存储到item对象(item类中进行相关属性的声明)
3.通过yild关键字将item提交到管道
4.管道文件中进行持久化存储代码的编写(process_item)
5.在配置文件中开启管道

chouti.py

# -*- coding: utf-8 -*-
import scrapy
import time
import re
from chouti.items import ChoutiItem


class ChoutinetSpider(scrapy.Spider):
    name = 'choutinet'
    # allowed_domains = ['www.chouti.com']
    start_urls = ['https://dig.chouti.com/']

    def parse(self, response):
        div_list = response.xpath('//div[@class="link-con"]/div')
        for div in div_list:
            title = div.xpath('.//a[@class="link-title link-statistics"]/text()').extract_first()
            content_link = div.xpath('.//a[@class="link-title link-statistics"]/@href').extract_first()
            content_link = self.format_link(content_link)
            author = div.xpath('.//span[@class="left author-name"]/text()').extract_first()
            time_point = div.xpath('.//span[@class="left time-enter timeago"]/span/@data-time').extract_first()
            ctime = self.format_time(time_point)
            # print(title, author, ctime, content_link)

            # 提交数据给管道
            item = ChoutiItem()
            item['title'] = title
            item['content_link'] = content_link
            item['author'] = author
            item['ctime'] = ctime
            yield item

    def format_time(self, time_point):
        """
        格式化时间，抓取的时间为字符串的时间戳，得加入.进行分割再转换
        """
        lis_time = list(str(time_point))
        lis_time.insert(10, ".")
        str_time = "".join(lis_time)
        ctime = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(float(str_time)))
        return ctime

    def format_link(self, content_link):
        """
        格式化标题链接，标题链接为未拼接链接，返回完整的链接
        """
        if re.search(r'/link/\d+', content_link):
            content_link = content_link.replace("/link", "link")
            total_link = self.start_urls[0] + content_link
        else:
            total_link = content_link
        return total_link

pipelines.py
注意:默认情况下,管道机制并没有开启.需要手动在配置文件中进行开启

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# 数据库配置
import pymongo
import redis
import json
import pymysql

class ChoutiPipeline:
    """
    保存到文件
    """
    fp = None

    # 只会被执行一次(开始爬虫的时候执行一次)
    def open_spider(self, spider):
        print('开始爬虫!!!')
        self.fp = open('./job.txt', 'w', encoding='utf-8')

    # 爬虫文件没提交一次item,该方法会被调用一次
    def process_item(self, item, spider):
        self.fp.write(item['title'] + "\n" + item['content_link'] + '\n' + item['author'] + '\n' + item['ctime'] + '\n')
        return item
        # [注意]一定要保证每一个管道类的process_item方法要有返回值

    def close_spider(self, spider):
        print('爬虫结束!!!')
        self.fp.close()


class ChoutiPipelineMongoDB:
    """
    保存到MongoDB数据库
    """

    def open_spider(self, spider):
        print('开始爬虫!!!')
        client = pymongo.MongoClient(host="127.0.0.1", port=27017)
        self.MONGO_DB = client["chouti"]

    def process_item(self, item, spider):
        dic = {
            "title": item['title'],
            "content_link": item['content_link'],
            "author": item['author'],
            "ctime": item['ctime'],
        }
        self.MONGO_DB.chouti_info.insert_one(dic)
        return item

    def close_spider(self, spider):
        print("爬虫结束")


class ChoutiPipelineRedis:
    """
    保存到Redis数据库中的list，需要进行对字典序列化和反序列化操作
    """
    redis_conn = None
    redis_db = 7

    def open_spider(self, spider):
        print('开始爬虫!!!')
        self.redis_conn = redis.StrictRedis(
            host="127.0.0.1", port=6379, db=self.redis_db, password="")

    def process_item(self, item, spider):
        dic = {
            "title": item['title'],
            "content_link": item['content_link'],
            "author": item['author'],
            "ctime": item['ctime'],
        }
        # print(self.redis_conn)
        # print(dic)
        self.redis_conn.lpush("chouti", json.dumps(dic))
        return item

    def close_spider(self, spider):
        res = self.redis_conn.lrange("chouti", 0, 0)
        # res返回值是一个列表
        print(json.loads(res[0]))
        print("爬虫结束")


class ChoutiPipelineMysql:
    """
    保存数据到 Mysql中，加入事件回滚操作
    """
    conn = None
    cursor = None

    def open_spider(self, spider):
        print('开始爬虫!!!')
        self.conn = pymysql.Connect(host='127.0.0.1', port=3306, user='root', password='root', db='spider')
        print(self.conn)

    def process_item(self, item, spider):
        self.cursor = self.conn.cursor()
        sql = 'insert into chouti values("%s","%s","%s","%s")' % (
            item['title'], item['content_link'], item['author'], item['ctime'])
        try:
            self.cursor.execute(sql)
            self.conn.commit()
        except Exception as e:
            print(e)
            self.conn.rollback()
        return item

    def close_spider(self, spider):
        print("爬虫结束")
        self.cursor.close()
        self.conn.close()

settings.py

# USER_AGENT = 'chouti (+http://www.yourdomain.com)'
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    'chouti.pipelines.ChoutiPipeline': 299,
    'chouti.pipelines.ChoutiPipelineMongoDB': 300,
    'chouti.pipelines.ChoutiPipelineRedis': 301,
    'chouti.pipelines.ChoutiPipelineMysql': 302,
}

items.py

import scrapy

class ChoutiItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    content_link = scrapy.Field()
    author = scrapy.Field()
    ctime = scrapy.Field()

3 图片分页多线程爬取（http://pic.netbian.com/4kmeinv/）

picnet.py

# -*- coding: utf-8 -*-
import scrapy
from picPro.items import PicproItem


class PicnetSpider(scrapy.Spider):
    name = 'picnet'
    # allowed_domains = ['http://pic.netbian.com/4kmeinv/']
    start_urls = ['http://pic.netbian.com/4kmeinv/']
    page = 1

    def parse(self, response):
        # print(response)
        li_list = response.xpath('//div[@class="slist"]/ul/li')
        # print(li_list)
        for li in li_list:
            img_url = "http://pic.netbian.com" + li.xpath('./a/img/@src').extract_first()
            img_name = li.xpath('./a/b/text()').extract_first()
            # print(img_url, img_name)
            item = PicproItem()
            item['img_name'] = img_name
            yield scrapy.Request(img_url, callback=self.get_img, meta={"item": item})

        # 爬取前10页图片
        if self.page < 11:
            self.page += 1
            print(self.page)
            new_url = self.start_urls[0] + "index_%d.html" % self.page
            yield scrapy.Request(new_url, callback=self.parse)

    def get_img(self, response):
        item = response.meta['item']
        item['img_data'] = response.body
        yield item

pipelines.py

import os
import time


class PicproPipeline:
    t = None

    def open_spider(self, spider):
        self.t = time.time()
        if not os.path.exists('picLib'):
            os.mkdir('./picLib')

    def process_item(self, item, spider):
        imgPath = './picLib/' + item['img_name'] + ".jpg"
        print(imgPath)
        with open(imgPath, 'wb') as fp:
            fp.write(item['img_data'])
            print(imgPath + '下载成功!')
        return item

    def close_spider(self, spider):
        print(time.time() - self.t)

settings.py

# 增加并发：CONCURRENT_REQUESTS 
#     默认scrapy开启的并发线程为32个，可以适当进行增加。在settings配置文件中修改CONCURRENT_REQUESTS = 100值为100,并发设置成了为100。
#
# 降低日志级别：LOG_LEVEL 
#     在运行scrapy时，会有大量日志信息的输出，为了减少CPU的使用率。可以设置log输出信息为INFO或者ERROR即可。在配置文件中编写：LOG_LEVEL = ‘INFO’
#
# 禁止cookie：COOKIES_ENABLED 
#     如果不是真的需要cookie，则在scrapy爬取数据时可以禁止cookie从而减少CPU的使用率，提升爬取效率。在配置文件中编写：COOKIES_ENABLED = False
#
# 禁止重试：RETRY_ENABLED 
#     对失败的HTTP进行重新请求（重试）会减慢爬取速度，因此可以禁止重试。在配置文件中编写：RETRY_ENABLED = False
#
# 减少下载超时：DOWNLOAD_TIMEOUT 
#     如果对一个非常慢的链接进行爬取，减少下载超时可以能让卡住的链接快速被放弃，从而提升效率。在配置文件中进行编写：DOWNLOAD_TIMEOUT = 10 超时时间为10s

CONCURRENT_REQUESTS = 8
LOG_LEVEL = 'ERROR'
COOKIES_ENABLED = False
RETRY_ENABLED = False
DOWNLOAD_TIMEOUT = 5

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False