Scrapy 添加代理
1、重写start_requests方法
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url,callback=self.parse, meta={'proxy':'http://127.0.0.1:7777'})
2、修改下载中间件,在process_requests方法中添加代理,或者重写DownloaderMiddleware类并添加process_request方法,最后在settings中启用DOWNLOADER_MIDDLEWARES
def process_request(self, request, spider):
request.meta['proxy'] = 'http://127.0.0.1:7777'
return None
启用:
DOWNLOADER_MIDDLEWARES = {
'xray_test.middlewares.XrayTestDownloaderMiddleware': 543,
}