scrapy中代理设置

Scrapy 添加代理

1、重写start_requests方法

def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(url,callback=self.parse, meta={'proxy':'http://127.0.0.1:7777'})

2、修改下载中间件,在process_requests方法中添加代理，或者重写DownloaderMiddleware类并添加process_request方法，最后在settings中启用DOWNLOADER_MIDDLEWARES

    def process_request(self, request, spider):
        request.meta['proxy'] = 'http://127.0.0.1:7777'
        return None

启用：

DOWNLOADER_MIDDLEWARES = {
   'xray_test.middlewares.XrayTestDownloaderMiddleware': 543,
}