1、在Middlewares.py中定义中间件:
#添加需要账号和密码身份验证的ip例子
import base64
class ProxyMiddleware(object):
def process_request(self,request,spider):
# 随机选出代理信息
proxy = "xxx.xxx.xxx.xxx:port"
# 设置代理的认证信息
auth = base64.b64encode(bytes("USERNAME:PASSWORD", 'utf-8'))
request.headers['Proxy-Authorization'] = b'Basic ' + auth
# 设置代理ip (http/https)
request.meta['proxy'] = 'http://' + proxy
2、在settings.py中开启中间件:
DOWNLOADER_MIDDLEWARES = {
'MySpider.middlewares.ProxyMiddleware': 200,
}
3、启动scrapy即可看到效果。