一、动态加载思路
使用selenium结合scrapy使用, 篡改响应数据, 得到动态加载的数据, 返回给spider
二、使用步骤
1. 使用selenium实例化浏览器对象
因为selenium实例化浏览器对象只需要被执行一次, 所以把创建浏览器对象的方法写到爬虫模块
from selenium import webdriver
class WangyiSpider(scrapy.Spider):
pro = webdriver.Chrome("D:\scWorkSpace\chromedriver.exe")
1.下载的middleware
from time import sleep
from scrapy import signals
# useful for handling different item types with a single interface
from itemadapter import is_item, ItemAdapter
from scrapy.http import HtmlResponse
class WangyiproDownloaderMiddleware:
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the downloader middleware does not modify the
# passed objects.
def process_request(self, request, spider):
return None
def process_response(self, request, response, spider):
# 获取在爬虫类中定义的浏览器对象
pro = spider.pro
if request.url in spider.models_urls:
# 基于selenium获取动态加载数据
pro.get(request.url)
sleep(3)
page_text = pro.page_source
new_response = HtmlResponse(url=request.url, body=page_text, encoding='utf-8', request=request)
return new_response
else:
return response
spider是爬虫对象, 可以通过spider.属性获取到爬虫模块创建的对象和属性.