python爬虫连接redis数据库

最新推荐文章于 2024-08-09 15:10:03 发布

heidyxlw

最新推荐文章于 2024-08-09 15:10:03 发布

阅读量373

点赞数

文章标签： redis 缓存 python

本文链接：https://blog.csdn.net/heidyxlw/article/details/126855991

版权

Redis 代理服务器存储管理 Python Scrapy

关键词由CSDN通过智能技术生成

定义一个redis_util类

from scrapy.utils.project import get_project_settings
import redis

class RedisUtil:
    r = {}
    def __init__(self):
        self.settings = get_project_settings()
        host = self.settings.get('REDIS_HOST')
        port = self.settings.get('REDIS_PORT')
        user = self.settings.get('REDIS_USER')
        password = self.settings.get('REDIS_PASSWORD')
        if len(RedisUtil.r.keys()) < 1:
            RedisUtil.r = redis.Redis(host=host, port=port, username=user, password=password, single_connection_client=True, decode_responses=True)

调用方法：

redis = RedisUtil()
if redis.r.get('proxy_host') == None:
    #获取代理地址
    try:
        proxy_host = get_proxy().get('ip')
        proxy_port = get_proxy().get('port')
        # 存入redis
        redis.r.set('proxy_host', proxy_host)
        redis.r.set('proxy_port', proxy_port)
    except Exception as e:
        raise e