crawlspider 命令
1.创建scrapy工程:scrapy startproject projectName 2.创建爬虫文件:scrapy genspider -t crawl spiderName www.xxx.com 指令多了 "-t crawl",表示创建的爬虫文件是基于CrawlSpider这个类的,而不再是Spider这个基类。
3.运行 scrapy crawl name --nolog
spider.py
class Spider2Spider(CrawlSpider): name = 'spider2' # allowed_domains = ['www.xxx.com'] start_urls = ['https://dig.chouti.com/r/scoff/hot/1'] rules = ( Rule(LinkExtractor(allow=r'/r/scoff/hot/\d+'), callback='parse_item', follow=True),
Rule(LinkExtractor(allow=r'/scoff/$'), callback='parse_item', follow=True),
) def parse_item(self, response): print(response)
scrapy-redis命令
运行命令:
cd scrapy2
cd spiders
scrapy runspider spider2.py
流程
1.创建scrapy工程:scrapy startproject projectName 2.创建爬虫文件:scrapy genspider -t crawl spiderName www.xxx.com 3.对爬虫文件中的相关属性进行修改: - 导包:from scrapy_redis.spiders import RedisCrawlSpider - 将当前爬虫文件的父类设置成RedisCrawlSpider - 将起始url列表替换成redis_key = 'xxx'(调度器队列的名称) - 注释掉start_urls = [] 4.在配置文件中进行配置: - 使用组件中封装好的可以被共享的管道类, 这个类在文件中看不到: ITEM_PIPELINES = { 'scrapy_redis.pipelines.RedisPipeline': 400 }
- 配置调度器(使用组件中封装好的可以被共享的调度器) # 增加了一个去重容器类的配置, 作用使用Redis的set集合来存储请求的指纹数据, 从而实现请求去重的持久化 DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" # 使用scrapy-redis组件自己的调度器 SCHEDULER = "scrapy_redis.scheduler.Scheduler" # 配置调度器是否要持久化, 也就是当爬虫结束了, 要不要清空Redis中请求队列和去重指纹的set。如果是True, 就表示要持久化存储, 就不清空数据, 否则清空数据 SCHEDULER_PERSIST = True - 指定存储数据的redis: REDIS_HOST = 'redis服务的ip地址' REDIS_PORT = 6379 - 配置redis数据库的配置文件 - 取消保护模式:protected-mode no - bind绑定: #bind 127.0.0.1 - 启动redis 5.执行分布式程序 scrapy runspider xxx.py 6.向调度器队列中仍入一个起始url: 在redis-cli中执行:
D:\program files\redis配置文件的配置:
- 注释该行:bind 127.0.0.1,表示可以让其他ip访问redis
- 将yes该为no:protected-mode no,表示可以让其他ip操作redis
spider2.py
# -*- coding: utf-8 -*- import scrapy from scrapy.linkextractors import LinkExtractor from scrapy.spiders import CrawlSpider, Rule from scrapy_redis.spiders import RedisCrawlSpider from scrapy2.items import Scrapy2Item class Spider2Spider(RedisCrawlSpider): name = 'spider2' # allowed_domains = ['www.xxx.com'] # start_urls = ['https://dig.chouti.com/r/scoff/hot/1'] redis_key = 'chouti' rules = ( Rule(LinkExtractor(allow=r'/all/hot/recent/\d+'), callback='parse_item', follow=True), ) def parse_item(self, response): div_list = response.xpath('//div[@class="item"]') for div in div_list: title = div.xpath('./div[4]/div[1]/a/text()').extract_first() author = div.xpath('./div[4]/div[2]/a[4]/b/text()').extract_first() item = Scrapy2Item() item['title'] = title item['author'] = author yield item
setttings.py
# -*- coding: utf-8 -*- # Scrapy settings for scrapy2 project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://doc.scrapy.org/en/latest/topics/settings.html # https://doc.scrapy.org/en/latest/topics/downloader-middleware.html # https://doc.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = 'scrapy2' SPIDER_MODULES = ['scrapy2.spiders'] NEWSPIDER_MODULE = 'scrapy2.spiders' USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'scrapy2 (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = False # Configure maximum concurrent requests performed by Scrapy (default: 16) CONCURRENT_REQUESTS = 32 # 增加了一个去重容器类的配置, 作用使用Redis的set集合来存储请求的指纹数据, 从而实现请求去重的持久化 DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" # 使用scrapy-redis组件自己的调度器 SCHEDULER = "scrapy_redis.scheduler.Scheduler" # 配置调度器是否要持久化, 也就是当爬虫结束了, 要不要清空Redis中请求队列和去重指纹的set。如果是True, 就表示要持久化存储, 就不清空数据, 否则清空数据 SCHEDULER_PERSIST = True #数据指纹 REDIS_HOST = '127.0.0.1' REDIS_PORT = 6379 ITEM_PIPELINES = { 'scrapy_redis.pipelines.RedisPipeline': 400 }
items.py
# -*- coding: utf-8 -*- # Define here the models for your scraped items # # See documentation in: # https://doc.scrapy.org/en/latest/topics/items.html import scrapy class Scrapy2Item(scrapy.Item): # define the fields for your item here like: title = scrapy.Field() author = scrapy.Field()
'movie_data',item