scrapy基础知识之随机切换fake-useragent 库的使用：

pip install fake-useragent

from fake_useragent import UserAgent
ua = UserAgent()

middlewares.py

 1 from fake_useragent import UserAgent
 2 
 3 class RandomUserAgent(object):
 4     # def __init__(self,crawl):
 5     #     super(RandomUserAgent,self).__init__()
 6     #     self.ua=UserAgent()
 7     def process_request(self, request, spider):
 8         #useragent = random.choice(USER_AGENTS)
 9         ua=UserAgent()
10         request.headers.setdefault("User-Agent",ua.random)

View Code

settints.py

1 DOWNLOADER_MIDDLEWARES = {
2       'LaGou.middlewares.RandomUserAgent': 1,
3 #    'LaGou.middlewares.MyCustomDownloaderMiddleware': 543,
4 }

View Code

 更多访问地址：https://github.com/hellysmile/fake-useragent

转载于:https://www.cnblogs.com/huwei934/p/6979286.html