Scrapy框架downloader middleware实现随机User-Agent设置

1.在middleware.py中添加一个新类：

class RandomUserAgentMiddleware():
    def __init__(self):
        # 随机的列表
        self.user_agents = ['User-Agent_01','User-Agent_02','...']
    def process_request(self,request,spider):
        request.headers['User-Agent'] = random.choice(self.user_agents)

2.在setting.py中取消注释downloader_middlewares，并修改为：

DOWNLOADER_MIDDLEWARES = {
   'scrapydownloadertest.middlewares.RandomUserAgentMiddleware': 543,
}