Scrapy-splash setting文件配置

最新推荐文章于 2023-09-27 09:27:17 发布

CodingCube

最新推荐文章于 2023-09-27 09:27:17 发布

阅读量225

点赞数 1

分类专栏： Python

本文链接：https://blog.csdn.net/m0_52440148/article/details/119833264

版权

Python 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

#渲染服务的URL
SPLASH_URL = 'http://192.168.99.100:8050'
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
#过滤器
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
#使用Splash的Http缓存
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
ROBOTSTXT_OBEY = False