scrapy加socks代理

最新推荐文章于 2025-03-06 15:53:13 发布

gu_zhi

最新推荐文章于 2025-03-06 15:53:13 发布

阅读量4.7k

点赞数 1

分类专栏：数据抓取文章标签： scrapy socks ip代理

本文链接：https://blog.csdn.net/Gu_zhi/article/details/87986269

版权

数据抓取专栏收录该内容

3 篇文章

订阅专栏

本文介绍了如何在Scrapy中配置SOCKS代理，通过自定义中间件`Socks5DownloadHandler`和`ScrapySocks5Agent`实现HTTP和HTTPS请求的代理下载。设置包括代理服务器地址、端口、用户名和密码。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

scrapy middlewares：

from txsocksx.http import SOCKS5Agent
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ClientEndpoint
from scrapy.core.downloader.webclient import _parse
from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler, ScrapyAgent

proxyHost = "socks-cla.abuyun.com"
proxyPort = xxxx
proxyUser = "S92IXXXXX2731G8C"
proxyPass = "D7650BABD8BXXXXX"

class Socks5DownloadHandler(HTTP11DownloadHandler):

def download_request(self, request, spider):
"""Return a deferred for the HTTP download"""
agent = ScrapySocks5Agent(contextFactory=self._contextFactory, pool=self._pool)
return agent.download_request(request)

class ScrapySocks5Agent(ScrapyAgent):

def _get_agent(self, request, timeout):
# bindAddress = request.meta.get('bindaddress') or self._bindAddress
#proxy = request.meta.get('proxy')
#if proxy:
#_, _, proxyHost, proxyPort, proxyParams = _parse(proxy)
#_, _, host, port, proxyParams = _parse(request.url)
torServerEndpoint = TCP4ClientEndpoint(reactor, proxyHost, proxyPort)
#proxyEndpoint = SOCKS5ClientEndpoint(proxyHost, proxyPort,proxyEndpoint=torServerEndpoint, methods={'login': (proxyUser, proxyPass)})

agent = SOCKS5Agent(reactor, proxyEndpoint=torServerEndpoint,endpointArgs=dict(methods={'login': (proxyUser, proxyPass)}))

return agent
# return self._Agent(reactor, contextFactory=self._contextFactory,
# connectTimeout=timeout, bindAddress=bindAddress, pool=self._pool)

scrapy settings：

DOWNLOAD_HANDLERS = {
"http": "spider.middlewares.Socks5DownloadHandler",
"https": "spider.middlewares.Socks5DownloadHandler",
}