单节点的redis太容易挂了,总是出现意外情况,借助scrapy-redis-sentinel库实现scrapy_redis对redis集群的使用
按照官方库的说明去修改后启动项目会报错,报错的原因其实是作者的源码包中一个叫queue.py的文件中PriorityQueue类有一些逻辑上的问题
1、首先安装库
pip3 install scrapy-redis-sentinel
2、安装后修改queue.py源码
pip3 show scrapy-redis-sentinel #查看源码包路径
进入包路径内
cd /usr/local/lib/python3.6/dist-packages/scrapy_redis_sentinel
修改queue.py中PriorityQueue类
class PriorityQueue(Base):
"""Per-spider priority queue abstraction using redis' sorted set"""
def __len__(self):
"""Return the length of the queue"""
return self.server.zcard(self.key)
def push(self, request):
"""Push a request"""
data = self._encode_request(request)
score = -request.priority
# We don't use zadd method as the order of arguments change depending on
# whether the class is Redis or StrictRedis, and the option of using
# kwargs only accepts strings, not bytes.
self.server.execute_command("ZADD", self.key, score, data)
def pop(self, timeout=0):
"""
Pop a request
timeout not support in this queue class
"""
if not isinstance(self.server, RedisCluster):
# use atomic range/remove using multi/exec
pipe = self.server.pipeline()
pipe.multi()
pipe.zrange(self.key, 0, 0).zremrangebyrank(self.key, 0, 0)
results, count = pipe.execute()
if results:
return self._decode_request(results[0])
# 使用集群的时候不能使用 multi/exec 来完成一个事务操作;使用lua脚本来实现类似功能
pop_lua_script = """
local result = redis.call('zrange', KEYS[1], 0, 0)
local element = result[1]
if element then
redis.call('zremrangebyrank', KEYS[1], 0, 0)
return element
else
return nil
end
"""
script = self.server.register_script(pop_lua_script)
results = script(keys=[self.key])
if results:
return self._decode_request(results)
修改完成后保存退出,其实就是把原来的else去掉了把if往后缩进了
3、修改settings.py配置文件,我就截图下需要修改的地方
4、修改spider文件
5、启动项目
scrapy_redis配置redis集群成功