爬虫 - Scrapy动态加载案例

---

一、动态加载思路

使用selenium结合scrapy使用, 篡改响应数据, 得到动态加载的数据, 返回给spider

二、使用步骤

1. 使用selenium实例化浏览器对象

因为selenium实例化浏览器对象只需要被执行一次, 所以把创建浏览器对象的方法写到爬虫模块

from selenium import  webdriver

class WangyiSpider(scrapy.Spider):
    pro = webdriver.Chrome("D:\scWorkSpace\chromedriver.exe")

1.下载的middleware

from time import sleep

from scrapy import signals

# useful for handling different item types with a single interface
from itemadapter import is_item, ItemAdapter
from scrapy.http import HtmlResponse


class WangyiproDownloaderMiddleware:
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    def process_request(self, request, spider):

        return None

    def process_response(self, request, response, spider):
    	# 获取在爬虫类中定义的浏览器对象
        pro = spider.pro
        if request.url in spider.models_urls:
        	# 基于selenium获取动态加载数据
            pro.get(request.url)
            sleep(3)
            page_text = pro.page_source
            new_response = HtmlResponse(url=request.url, body=page_text, encoding='utf-8', request=request)
            return new_response
        else:
            return response

spider是爬虫对象, 可以通过spider.属性获取到爬虫模块创建的对象和属性.