scrapy添加讯代理动态转发（Python3）

最新推荐文章于 2024-06-13 14:47:28 发布

chenxunan5025

最新推荐文章于 2024-06-13 14:47:28 发布

阅读量818

点赞数 1

文章标签： python

原文链接：https://my.oschina.net/chenmoxuan/blog/3095926

版权

1. 先行条件

将scrapy源代码，相对路径为：Lib/site-packages/scrapy/core/downloader/handlers/http11.py的文件中，

if isinstance(agent, self._TunnelingAgent):
   headers.removeHeader(b'Proxy-Authorization')

给注释掉。

必须将此注释掉，否则proxy-authorization会被去除，动态转发失效。

2. 示例middleware

class EpDownloaderMiddleware(object):
    def __init__(self):
        self.orderno = "XXXXXXXXXXXXXXXXXXXXXXX"
        self.secret = "XXXXXXXXXXXXXXXXXXXXXXX"

    def process_request(self, request, spider):
        request.meta['proxy'] = 'http://forward.xdaili.cn:80'
        timestamp = str(int(time.time()))  # timestamp
        string = "orderno=" + self.orderno + "," + "secret=" + self.secret + "," + "timestamp=" + timestamp
        md5_string = hashlib.md5(string.encode('utf-8')).hexdigest()  # sign
        sign = md5_string.upper()  
        auth = "sign=" + sign + "&" + "orderno=" + self.orderno + "&" + "timestamp=" + timestamp
        request.headers["Proxy-Authorization"] = auth

转载于:https://my.oschina.net/chenmoxuan/blog/3095926