Scrapy第九篇：User-Agent中间件

1.安装依赖

pip install fake-useragent -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install -U fake-useragent -i https://pypi.tuna.tsinghua.edu.cn/simple

2.编写中间件：打开middlewares.py文件，新建中间件RandomUserAgentMidddlware

from fake_useragent import UserAgent


class RandomUserAgentMidddlware(object):
    def __init__(self, crawler):
        super(RandomUserAgentMidddlware, self).__init__()
        self.ua = UserAgent()
        self.ua_type = crawler.settings.get('RANDOM_UA_TYPE', 'random')

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_request(self, request, spider):
        request.headers['User-Agent'] = getattr(self.ua, self.ua_type)

3.开启中间件：编辑settings.py。

DOWNLOADER_MIDDLEWARES = {
    'announcement.middlewares.RandomUserAgentMidddlware': 542,
}

4.验证一下 

5.如果使用报错：参考

fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached_苍穹之跃的博客-CSDN博客pip install -U fake-useragenthttps://blog.csdn.net/wenxingchen/article/details/118164460