scrapy中使用fake-useragent库设置user-agent

安装fake-useragent库(useragent的库)

pip install fake-useragent

在middlewares.py文件中导入fake-useragent库

from fake_useragent import UserAgent

在middlewares.py文件中编写一个middleware中间件，

from fake_useragent import UserAgent
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware

class MyUserAgentMiddleware(UserAgentMiddleware):
    def __init__(self, user_agent=''):
        self.user_agent = user_agent

    def process_request(self, request, spider): 
        ua = UserAgent()
        request.headers['User-Agent'] = ua.random

在setting.py文件中启用DOWNLOADER_MIDDLEWARES，并且把默认的注释掉

DOWNLOADER_MIDDLEWARES = {
    #'amazon.middlewares.MyCustomDownloaderMiddleware': 543,
    'amazon.middlewares.MyUserAgentMiddleware': 543,
}

更多：https://blog.51cto.com/laoyinga/2046970