scrapy中使用讯代理转发

最新推荐文章于 2024-03-18 10:30:25 发布

九成N

最新推荐文章于 2024-03-18 10:30:25 发布

阅读量295

点赞数

分类专栏：爬虫

本文链接：https://blog.csdn.net/mz02230909mz/article/details/107933071

版权

爬虫专栏收录该内容

14 篇文章 1 订阅

订阅专栏

scrapy源代码中查找http11.py文件，相对路径为：
Lib/site-packages/scrapy/core/downloader/handlers/http11.py

找到下面内容，注释掉：
if isinstance(agent, self._TunnelingAgent):
headers.removeHeader(b’Proxy-Authorization’)

否则proxy-authorization会被去除，动态转发失效。

自定义下载中间件：
class ProxyIPMiddleware(object):
‘’’
随机更换代理ip
‘’’
def init(self):
self.orderno = “xxxxxxxxxxxx” # 订单号
self.secret = “xxxxxxxxxxx” # 秘钥

def process_request(self,request,spider):
    print('====ProxyIPMiddleware====')
    protocal = request.url.split(':')[0].strip().lower()
    print(request.url,'protocal:',protocal)
    ip = "forward.xdaili.cn"  # 代理ip
    port = "80"  # 端口号
    ip_port = ip + ":" + port
    proxy = {"http": "http://" + ip_port, "https": "https://" + ip_port}
    timestamp = str(int(time.time()))  # 时间戳
    string = "orderno=" + self.orderno + "," + "secret=" + self.secret + "," + "timestamp=" + timestamp
    md5_string = hashlib.md5(string.encode()).hexdigest()  # md5哈希，得到固定长度的字符串
    sign = md5_string.upper()  # 转换成大写字母
    # 认证信息
    auth = "sign=" + sign + "&" + "orderno=" + self.orderno + "&" + "timestamp=" + timestamp
    print('auth:',auth)
    request.headers['Proxy-Authorization'] = auth
    #HTTP代理，只代理HTTP网站，HTTPS代理，只代理HTTPS网站
    request.meta['proxy'] = proxy[protocal]