文章目录
一. CrawlSpider
现实情况下,我们需要对满足某个特定条件的url进行爬取,这时候就可以通过CrawlSpider完成。
CrawlSpider继承自Spider,只不过在之前的基础上增加了新的功能,可以定义爬取的url规则,Scrapy碰到满足条件的url都进行爬取,而不用手动的yield Request。
二. CrawlSpider案例
1. 目录结构
2. wxapp_spider.py
# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from wxapp.items import WxappItem
class WxappSpiderSpider(CrawlSpider):
name = 'wxapp_spider'
allowed_domains = ['wxapp-union.com']
start_urls = ['http://www.wxapp-union.com/portal.php?mod=list&catid=1&page=1']
rules = (
Rule(LinkExtractor(allow=r'.+mod=list&catid=1&page=\d'), follow=True),
Rule(LinkExtractor(allow=r'.+article-.+\.html'), callback='parse_detail', follow=False)
)
def parse_detail(self, response):
title = response.xpath