scrapy中user-agent（用户代理）的“随机更换”功能

最新推荐文章于 2021-03-26 16:47:58 发布

beholder_lin

最新推荐文章于 2021-03-26 16:47:58 发布

阅读量358

点赞数

本文链接：https://blog.csdn.net/beholder_lin/article/details/104379957

版权

scrapy中use-ragent（用户代理）的“随机更换”功能
一.采用random模块并在middlewares中将所用的user-agent列出
二.利用fake_agent第三方库实现user-agent的随机更换
一.
1.在middlewares中导入randomimport random
2.新建一个类并将所用的user-agent列出

class RandomUserAgentMiddlemares(object):
#随便在网上找的
    agents = [
 "Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; Nexus S Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
 "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)",
 "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.310.0 Safari/532.9",
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7",
 "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.601.0 Safari/534.14",
 "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14",
 "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20",
 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.27 (KHTML, like Gecko) Chrome/12.0.712.0 Safari/534.27",
 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.24 Safari/535.1",
 "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2",
 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7",
 "Mozilla/5.0 (Windows; U; Windows NT 6.0 x64; en-US; rv:1.9pre) Gecko/2008072421 Minefield/3.0.2pre",
 "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10",
 "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)",
 "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB5",
 "Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0E)",
 "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
 "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
 "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0",]
#————————————————
#版权声明：本文为CSDN博主「dasny斑马」的原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接及本声明。
#原文链接：https://blog.csdn.net/u013039658/article/details/80673470
    def process_request(self,spider,request):
        user_agent=random.choice(self.agents)
        request.headers["User-Agent"]=user_agent

3.在settings中打开downloadmiddlewares并完成其他配置信息

DOWNLOADER_MIDDLEWARES = {
    'work.middlewares.RandomUserAgentMiddlemares': 543,
}

二.
1.下载fake_useragent

pip install fake_useragent

2.在middlewares中导入fake_useragent并新建一个类

from fake_useragent import UserAgent
class RandomUserAgentMiddleware(object):
    #随机更换user-agent
    def __init__(self,crawler):
        super(RandomUserAgentMiddleware,self).__init__()
        self.ua = UserAgent()
        self.ua_type = crawler.settings.get("RANDOM_UA_TYPE","random")
 
    @classmethod
    def from_crawler(cls,crawler):
        return cls(crawler)
 
    def process_request(self,request,spider):
        def get_ua():
            return getattr(self.ua,self.ua_type)
 
        request.headers.setdefault('User-Agent',get_ua())

3.在settings中进行配置并定义RANDOM_UA_TYPE=“random”（“random”可以为“chrome”，“firefox”等等，用于生成不同浏览器的User-Agent）

RANDOM_UA_TYPE= 'random'
DOWNLOADER_MIDDLEWARES = {
    'suiyi.middlewares.RandomUserAgentMiddleware': 543,
    # 导入系统自带的useragent类，并将优先级设置为none
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}