爬虫必备良药（一）——设置随机User-Agent

前言

最近在自学爬虫，琢磨各种防封、防反爬虫措施，于是乎想自己收集一些user-agent用来做随机header，偶然发现github上有一个专门提供随机ua的库，遂记录之

GitHub

名称：fake-useragent

Demo

setting.py

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'ArticleSpider.middlewares.RandomUserAgentMiddleware': 543,
}
RANDOM_UA_TYPE="random"

middlewares.py

from fake_useragent import UserAgent

#####################################
    #这里都是scrapy自动生成的代码#
#####################################


class RandomUserAgentMiddleware(object):
    #设置随机User-Agent

    def __init__(self, crawler):
        super(RandomUserAgentMiddleware,self).__init__()
        self.ua=UserAgent()