scrapy加socks代理

最新推荐文章于 2024-08-29 20:07:11 发布

gu_zhi

最新推荐文章于 2024-08-29 20:07:11 发布

阅读量4.5k

点赞数 1

分类专栏：数据抓取文章标签： scrapy socks ip代理

本文链接：https://blog.csdn.net/Gu_zhi/article/details/87986269

版权

本文介绍了如何在Scrapy中配置SOCKS代理，通过自定义中间件`Socks5DownloadHandler`和`ScrapySocks5Agent`实现HTTP和HTTPS请求的代理下载。设置包括代理服务器地址、端口、用户名和密码。

摘要由CSDN通过智能技术生成

scrapy middlewares：

from txsocksx.http import SOCKS5Agent
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ClientEndpoint
from scrapy.core.downloader.webclient import _parse
from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler, ScrapyAgent

proxyHost = "socks-cla.abuyun.com"
proxyPort = xxxx
proxyUser = "S92IXXXXX2731G8C"
proxyPass = "D7650BABD8BXXXXX"

class Socks5DownloadHandler(HTTP11DownloadHandler):

def download_request(self, request, spider):
"""Return a deferred for the HTTP download"""
agent = ScrapySocks5Agent(contextFactory=self._contextFactory, pool=self._pool)
return agent.download_request(request)

class ScrapySocks5Agent(ScrapyAgent):

def _get_agent(self, request, timeout):
# bindAddress &#