scrapy的反反爬

爬虫的反反爬策略

® 反反爬的总体思想
	○ 将爬虫伪装成自然人的浏览行为
® 自然人浏览的特点
	○ 访问频率不会太过于高
	○ 使用浏览器进行访问
	○ 网站设置登录要求后仍然能够正常访问
	○ 可以完成验证操作
® scrapy的反反爬功能的实现
	○ 爬虫配置文件setting.py
	○ 爬虫框架的中间件

使用随机的ip

解决封锁IP的反爬机制

首先要在setting中设置代理IP池(要确保我们所使用代理IP可用有效！！！)

IPPools = [{'ipaddr': '120.55.241.70:80'},
           {'ipaddr': '115.62.183.165:8118 '},
           {'ipaddr': '183.154.208.229:8085 '},
           {'ipaddr':'112.35.204.111:80'}
           ]

然后在middlewares.py文件中的process_request方法中是同random选取一个随机的IP，写入到request的meta属性

    def process_request(self, request, spider):
        thisip = random.choice(IPPools)
        request.meta['proxy'] = "https://" + thisip['ipaddr']
        print("this is ip:%s" % thisip['ipaddr'])

再次回到setting.py文件中注册并且设置优先级

DOWNLOADER_MIDDLEWARES = {
    "ip_broker.middlewares.IpBrokerDownloaderMiddleware": 123,
    }

最后打开spider的爬虫文件在response中获取到meta属性中的proxy的值

print(response.meta['proxy'])

最后一步是为了验证IP是不是真的转换为你写的代理IP

设置随机的user_agent

为了解决对user_agnet的封锁
创建一个类并且将user_agent的请求头都放到列表中

class UserAgentMiddleware(object):
    def __init__(self):
        self.user_agent_list = [
            "Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/39.0.2171.95Safari/537.36OPR/26.0.1656.60"
            "Opera/8.0(WindowsNT5.1;U;en)",
            "Mozilla/5.0(WindowsNT5.1;U;en;rv:1.8.1)Gecko/20061208Firefox/2.0.0Opera9.50",
            "Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1;en)Opera9.50",
            "Mozilla/5.0(WindowsNT6.1;WOW64;rv:34.0)Gecko/20100101Firefox/34.0",
            "Mozilla/5.0(X11;U;Linuxx86_64;zh-CN;rv:1.9.2.10)Gecko/20100922Ubuntu/10.10(maverick)Firefox/3.6.10",
            "Mozilla/5.0(Macintosh;U;IntelMacOSX10_6_8;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50",
            "Mozilla/5.0(Windows;U;WindowsNT6.1;en-us)AppleWebKit/534.50(KHTML,likeGecko)Version/5.1Safari/534.50",
            "Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/534.57.2(KHTML,likeGecko)Version/5.1.7Safari/534.57.2",
            "Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/39.0.2171.71Safari/537.36",
            "Mozilla/5.0(X11;Linuxx86_64)AppleWebKit/537.11(KHTML,likeGecko)Chrome/23.0.1271.64Safari/537.11",
            "Mozilla/5.0(Windows;U;WindowsNT6.1;en-US)AppleWebKit/534.16(KHTML,likeGecko)Chrome/10.0.648.133Safari/534.16",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Maxthon2.0)",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TencentTraveler4.0)",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;TheWorld)",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Trident/4.0;SE2.XMetaSr1.0;SE2.XMetaSr1.0;.NETCLR2.0.50727;SE2.XMetaSr1.0)",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;360SE)",
            "Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/30.0.1599.101Safari/537.36",
            "Mozilla/5.0(WindowsNT6.1;WOW64;Trident/7.0;rv:11.0)likeGecko",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;AvantBrowser)",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)",
            "Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Trident/5.0;)",
            "Mozilla/4.0(compatible;MSIE8.0;WindowsNT6.0;Trident/4.0)",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT6.0)",
            "Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1)",
            'Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/536.11(KHTML,likeGecko)Chrome/20.0.1132.11TaoBrowser/2.0Safari/536.11',
            "Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/537.1(KHTML,likeGecko)Chrome/21.0.1180.71Safari/537.1LBBROWSER",
            "Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;WOW64;Trident/5.0;SLCC2;.NETCLR2.0.50727;.NETCLR3.5.30729;.NETCLR3.0.30729;MediaCenterPC6.0;.NET4.0C;.NET4.0E;LBBROWSER)",
            "Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1;SV1;QQDownload732;.NET4.0C;.NET4.0E;LBBROWSER)",
            "Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;WOW64;Trident/5.0;SLCC2;.NETCLR2.0.50727;.NETCLR3.5.30729;.NETCLR3.0.30729;MediaCenterPC6.0;.NET4.0C;.NET4.0E;QQBrowser/7.0.3698.400)",
            "Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1;SV1;QQDownload732;.NET4.0C;.NET4.0E)",
            "Mozilla/5.0(WindowsNT5.1)AppleWebKit/535.11(KHTML,likeGecko)Chrome/17.0.963.84Safari/535.11SE2.XMetaSr1.0",
            "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Trident/4.0;SV1;QQDownload732;.NET4.0C;.NET4.0E;SE2.XMetaSr1.0)",
            "Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Maxthon/4.4.3.4000Chrome/30.0.1599.101Safari/537.36",
            "Mozilla/5.0(WindowsNT6.1;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/38.0.2125.122UBrowser/4.0.3214.0Safari/537.36",
        ]

重写process_request方法

 def process_request(self, request, spider):
        user_agent = random.choice(self.user_agent_list)
        request.headers['User-Agent'] = user_agent
        print('user_agent:%s' % user_agent)

最后在setting.py文件中注册并且设置优先级

 "ip_broker.middlewares.UserAgentMiddleware": 123

这样可以解决对于封锁ip和封锁user_agent的反爬虫。