使用CrawlSpider爬取糗事百科段子

CrawlSpider深度爬取

CrawlSpider是什么:

crawlspider也是一个spider,是spider的一个子类，所以其功能要比Spider要强大。
多的功能是：提取链接的功能，根据一定的规则，提取指定的链接。

链接提取器：

LinkExtractor(
	allow = xxx, # 正则表达式,要(*)
	deny = xxx, # 正则表达式，不要这个
	restrict_xpaths = xxx, # xpath路径(*)
	restrict_css = xxx, # 选择器(*)
	deny_domains = xxx, # 不允许的域名
	)

项目截图：

运行命令：

1. scrapy startproject news
2. cd news
3. scrapy genspider -t crawl qiubai www.qiushibaike.com

示例代码：

items.py文件中:

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class NewsItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # 用户头像的url地址
    icon_url = scrapy.Field()
    # 用户名
    username = scrapy.Field()
    # 用户年龄
    age = scrapy.Field()
    # 用户发表的内容
    content = scrapy.Field()
    # 好笑的个数
    haha_count = scrapy.Field()
    # 评论数量
    coment_count = scrapy.Field()

settings.py文件：

# Scrapy settings for news project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'news'
LOG_LEVEL = 'ERROR'
SPIDER_MODULES = ['news.spiders']
NEWSPIDER_MODULE = 'news.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = '改为自己的User-Agent'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0.6
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'news.middlewares.NewsSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'news.middlewares.NewsDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'news.pipelines.NewsPipeline': 300,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

piplines.py文件中:

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
import json

from itemadapter import ItemAdapter


class NewsPipeline:
    # 重写这个方法，当爬虫开启的时候就会调用这个方法
    def open_spider(self, spider):
        self.fp = open('qiubai.txt', 'w', encoding='utf8')

    # 处理item数据的方法
    def process_item(self, item, spider):
        # 要将item保存到文件中
        # 将对象转化为字典
        dic = dict(item)
        # 将字典转化为json数据
        strin = json.dumps(dic, ensure_ascii=False)
        self.fp.write(strin + '\n')
        return item

    # 当爬虫结束时候调用这个方法
    def close_spider(self, spider):
        self.fp.close()

qiubai.py文件中:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule

from news.items import NewsItem


class QiubaiSpider(CrawlSpider):
    name = 'qiubai'
    # allowed_domains = ['www.qiushibaike.com']
    start_urls = ['https://www.qiushibaike.com/text/']
	# 根据规则提取链接
    rules = (
    	# Rule(LinkExtractor(allow=r''), callback='parse_item', follow=True)如果是这样，会提取起始url这个页面以下的所有链接
        Rule(LinkExtractor(allow=r'/text/page/\d+/'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):

        #item['domain_id'] = response.xpath('//input[@id="sid"]/@value').get()
        #item['name'] = response.xpath('//div[@id="name"]').get()
        #item['description'] = response.xpath('//div[@id="description"]').get()
        content_div = response.xpath('//*[@id="content"]/div/div[2]/div')
        for content_d in content_div:
            item = NewsItem()
            # 头像的url地址
            icon_url = content_d.xpath('.//div/a/img/@src').extract_first()
            icon_url = 'https:' + icon_url
            # 用户名
            username = content_d.xpath('.//div/a[2]/h2/text()').extract_first().strip('\n')
            # 年龄
            age = content_d.xpath('.//div/div/text()').extract_first()
            # 内容 //*[@id="qiushi_tag_124466562"]/a[1]/div/span/text()[2]
            content = content_d.xpath('.//a[1]/div[@class="content"]/span[1]').xpath('string(.)').extract_first()
            # 好笑个数
            haha_count = content_d.xpath('.//div[2]/span[1]/i/text()').extract_first()
            # 评论个数
            comment_count = content_d.xpath('.//div[2]/span[2]/a/i/text()').extract_first()
            item['icon_url'] = icon_url
            item['username'] = username
            item['age'] = age
            item['content'] = content.strip('\n')
            item['haha_count'] = haha_count
            item['coment_count'] = comment_count
            yield item

链接提取器不管用什么方式提取链接，都会把重复的链接自动去重。在scrapy shell中，可以这样：

link = LinkExtractor(allow=r'/text/page/\d+/')
link.extract_links(response) # 进行查看提取的链接

注意：

1. 一个链接提取器对应一个规则解析器，多个链接提取器对应多个规则解析器。

link1 = LinkExtractor(allow=r'/text/page/\d+/')
link2 = LinkExtractor(allow=r'/text/page/\d+/')
  rules = (
    	Rule(link1, callback='parse_item', follow=True)
        Rule(link2, callback='parse_item', follow=True),
    )

3. 在实现深度爬取的过程中需要和scrapy.Request()结合使用。