根据scrapy官方文档:http://doc.scrapy.org/en/master/topics/practices.html#avoiding-getting-banned里面的描述,要防止scrapy被ban,主要有以下几个策略。
1.动态设置user agent
2.禁用cookies
3.设置延迟下载
4.使用IP地址池(Tor project、VPN和代理IP)
5.使用Crawlera
1、创建middlewares.py
1.动态设置user agent
2.禁用cookies
3.设置延迟下载
4.使用IP地址池(Tor project、VPN和代理IP)
5.使用Crawlera
1、创建middlewares.py
scrapy代理IP、user agent的切换都是通过DOWNLOADER_MIDDLEWARES进行控制,下面我们创建middlewares.py文件。
<span style="font-size:14px;">import random
import base64
from settings import PROXIES
class RandomUserAgent(object):
"""Randomly rotate user agents based on a list of predefined ones"""
def __init__(self, agents):
self.agents = agents
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings.getlist('USER_AGENTS'))
def process_request(self, request, spider):
#print "**************************" + random.choice(self.agents)
request.headers.setdefault('User-Agent', random.choice(self.agents))
class ProxyMiddleware(object):
def process_request(self, request, spider):
proxy = random.choice(PROXIES)
if proxy['user_pass'] is not None:
request.meta['proxy'] = "http://%s" % proxy['ip_port']
encoded_user_pass = base64.encodestring(proxy['user_pass'])
request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
print "**************ProxyMiddleware have pass************&