Scrapy(Scrapy+Scrapy-splash)抓取js动态页面入门笔记

本文章仅作为个人笔记

还不懂Scrapy的可以参考

Scrpy官网

Scrpy官方文档

Scrpy中文文档

Scrpy-splash项目git地址

个人ScrapyDemo项目地址

准备工作
  • 先完成简单scrapy项目
  • 安装docker

    • win下下载安装包安装
    • mac下下载安装包安装(尝试使用brew安装,安装启动过程非常复杂,最后选择使用安装包直接安装)
    • centos7下运行:

      yum install docker

  • redhat运行:

    yum install --setopt=obsoletes=0 docker-ce-17.03.2.ce-1.el7.centos.x86_64 docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch
    
  • 安装 scrapy-splash

    pip install scrapy-splash
    
  • 启动docker服务

    • centos7

      service docker start

    • win下直接打开应用

    • mac下直接打开应用
  • 拉取镜像

    docker pull scrapinghub/splash
    
  • 运行镜像

    docker run -p 8050:8050 scrapinghub/splash
    
  • 配置splash服务(以下操作全部在settings.py):

    • 添加splash服务器地址:

      SPLASH_URL = ‘http://localhost:8050

    • 将splash middleware添加到DOWNLOADER_MIDDLEWARE中:

      DOWNLOADER_MIDDLEWARES = {
          'scrapy_splash.SplashCookiesMiddleware': 723,
          'scrapy_splash.SplashMiddleware': 725,
          'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
      }
      
    • Enable SplashDeduplicateArgsMiddleware:

      SPIDER_MIDDLEWARES = {
          'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
      }
      
    • Set a custom DUPEFILTER_CLASS:

      DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
      
    • a custom cache storage backend:

      HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
      
  • 例子

    import json, scrapy
    
    lass MySpider(scrapy.Spider):
       name = 'example'
       allowed_domains = ['example.com']
       start_urls = ["http://example.com", "http://example.com/foo"]
    
       def start_requests(self):
         for url in self.start_urls:
           yield SplashRequest(url, self.parse, args={'wait': 0.5})
    
       def parse(self, response):
           # ...
    
©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页