一、问题描述
scrapy-redis中使用RedisCrawlSpider类爬虫,以Rule规则来匹配地址,运行爬虫时出现错误:
[scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'XXX'
二、解决
在settings.py中添加
SPIDER_MIDDLEWARES = { 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None, }
原因:
Don’t cry because it is over, smile because it happened.