解决 Kubernetes 部署 Redis Sentinel 本地无法连接的问题

coding....

已于 2023-07-27 14:45:13 修改

阅读量693

点赞数

文章标签： kubernetes redis sentinel spring boot

于 2023-07-27 14:32:43 首次发布

本文链接：https://blog.csdn.net/qq_41896122/article/details/131957405

版权

解决 Kubernetes 部署 Redis Sentinel 本地无法连接的问题

问题：在 kubernetes 部署 redis sentinel 时无法直接通过哨兵访问redis，因为哨兵返回的redis地址是pod ip或者是 service name，本地无法直接访问 pod ip，所以无法访问。

解决方案：

如果哨兵返回的是 service name
可以直接配置本机 hosts文件，将 service name 路由至 redis master 节点的 nodePort, 这种可以解决问题，但不是长久之计。
目前我能想到的是开发环境使用单点连接 redis 的方式，在测试/生产环境使用 sentinel或者cluster 模式
这种方式也有问题，我来说下问题。
首先我们使用Spring Boot 连接 redis 的方式 maven 引用 starter

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-redis</artifactId>
		</dependency>

这是官方推荐我们集成的方式，简单，配置方便，支持单点，哨兵，集群三种模式，只需要在配置文件中配置就好,例如这样

spring:
  application:
    name: kubernetes-apiserver
  mvc:
    pathmatch:
      matching-strategy: ant_path_matcher
  redis:
    connect-type: sentinel
    client-type: lettuce
    database: 0
    host: 127.0.0.1
    port: 6379
    password: ******
    lettuce:
      pool:
        max-active: 50
        max-wait: 10ms
        max-idle: 10
        min-idle: 0
        enabled: true
    sentinel:
      master: mymaster
      password: *******
      nodes: 
      - *****
      - *****
      - *****
    cluster:
      nodes:       
      - *****
      - *****
      - *****

但是这种方式在我这种情况的缺点在于如果配置了哨兵直接就用了哨兵的配置，他的优先级是 哨兵 > 集群 > 单点。
配置文件一定是不能总去更改的，一定是都配置上的。那这样开发启动的时候就会连哨兵，这样还是连不上。所以我决定重写这块连接的逻辑，根据配置字段来判断连接哨兵，单点，或是集群。
所以这里我添加了个配置值 spring.redis.connect-type。通过这个字段来判断连接什么。
接下来来改造连接方式：

首先编写个枚举存储三种类型

@Getter
@AllArgsConstructor
public enum EnumRedisConnectType {

    SINGLE("single") ,
    SENTINEL("sentinel"),
    CLUSTER("cluster");

    private final String type;

}

通过查看Redis配置源码可以看见RedisConnectionConfiguration 有两个子类，分别对应两种连接池 Jedis，Lettuce，这里注意就是 Jedis 模式需要引用相关依赖包，Spring Data Redis 默认是 Lettuce 模式。可以通过Spring.redis.client-type 进行配置。

因为该类不是 public 的，所以我们需要重写这类和相关的两种实现。这里我们直接复制到代码目录下即可。
涉及到的类有:
RedisUrlSyntaxException 异常类
RedisConnectionConfiguration
LettuceConnectionConfiguration
JedisConnectionConfiguration

复制好后更改下类中涉及到的类包，让代码不再报错。注意这里需要引用除了 pring-boot-starter-data-redis 以外的两个依赖包，分别是apache的 common-pool2 和 jedis

		<dependency>
			<groupId>redis.clients</groupId>
			<artifactId>jedis</artifactId>
			<version>3.8.0</version>
		</dependency>

		<dependency>
			<groupId>org.apache.commons</groupId>
			<artifactId>commons-pool2</artifactId>
			<version>2.11.1</version>
		</dependency>

接下来展示下如何改造代码，拿LettuceConnectionConfiguration 举例，Jedis 的同理。这里只改造了这个方法：createLettuceConnectionFactory()，非常简单。Jedis 的就不进行粘贴了。

package com.coding.apiserver.config;

import io.lettuce.core.ClientOptions;
import io.lettuce.core.RedisClient;
import io.lettuce.core.SocketOptions;
import io.lettuce.core.TimeoutOptions;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.ClusterTopologyRefreshOptions;
import io.lettuce.core.resource.ClientResources;
import io.lettuce.core.resource.DefaultClientResources;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.autoconfigure.data.redis.ClientResourcesBuilderCustomizer;
import org.springframework.boot.autoconfigure.data.redis.LettuceClientConfigurationBuilderCustomizer;

import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.RedisSentinelConfiguration;
import org.springframework.data.redis.connection.RedisStandaloneConfiguration;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettucePoolingClientConfiguration;
import org.springframework.util.StringUtils;

import java.time.Duration;

@Configuration(proxyBeanMethods = false)
@ConditionalOnClass(RedisClient.class)
@ConditionalOnProperty(name = "spring.redis.client-type", havingValue = "lettuce", matchIfMissing = true)
public class LettuceConnectionConfiguration extends RedisConnectionConfiguration {

    //这里将配置文件的配置连接类型引用过来，single/sentinel/cluster
    @Value("${spring.redis.connect-type}")
    private String connectType;

    LettuceConnectionConfiguration(RedisProperties properties,
                                   ObjectProvider<RedisStandaloneConfiguration> standaloneConfigurationProvider,
                                   ObjectProvider<RedisSentinelConfiguration> sentinelConfigurationProvider,
                                   ObjectProvider<RedisClusterConfiguration> clusterConfigurationProvider) {
        super(properties, standaloneConfigurationProvider, sentinelConfigurationProvider, clusterConfigurationProvider);
    }

    @Bean(destroyMethod = "shutdown")
    @ConditionalOnMissingBean(ClientResources.class)
    DefaultClientResources lettuceClientResources(ObjectProvider<ClientResourcesBuilderCustomizer> customizers) {
        DefaultClientResources.Builder builder = DefaultClientResources.builder();
        customizers.orderedStream().forEach((customizer) -> customizer.customize(builder));
        return builder.build();
    }

    @Bean
    @ConditionalOnMissingBean(RedisConnectionFactory.class)
    LettuceConnectionFactory redisConnectionFactory(
            ObjectProvider<LettuceClientConfigurationBuilderCustomizer> builderCustomizers,
            ClientResources clientResources) {
        LettuceClientConfiguration clientConfig = getLettuceClientConfiguration(builderCustomizers, clientResources,
                getProperties().getLettuce().getPool());
        return createLettuceConnectionFactory(clientConfig);
    }

	//这里就是改造后的代码，判断connectType 创建工厂类即可，Jedis 同理更改，就不进行粘贴了
    private LettuceConnectionFactory createLettuceConnectionFactory(LettuceClientConfiguration clientConfiguration) {
        //默认单点连接
        if (EnumRedisConnectType.SENTINEL.getType().equals(connectType)) {
            return new LettuceConnectionFactory(getSentinelConfig(), clientConfiguration);
        }else if (EnumRedisConnectType.CLUSTER.getType().equals(connectType)) {
            return new LettuceConnectionFactory(getClusterConfiguration(), clientConfiguration);
        }else {
            return new LettuceConnectionFactory(getStandaloneConfig(), clientConfiguration);
        }
    }

    private LettuceClientConfiguration getLettuceClientConfiguration(
            ObjectProvider<LettuceClientConfigurationBuilderCustomizer> builderCustomizers,
            ClientResources clientResources, RedisProperties.Pool pool) {
        LettuceClientConfiguration.LettuceClientConfigurationBuilder builder = createBuilder(pool);
        applyProperties(builder);
        if (StringUtils.hasText(getProperties().getUrl())) {
            customizeConfigurationFromUrl(builder);
        }
        builder.clientOptions(createClientOptions());
        builder.clientResources(clientResources);
        builderCustomizers.orderedStream().forEach((customizer) -> customizer.customize(builder));
        return builder.build();
    }

    private LettuceClientConfiguration.LettuceClientConfigurationBuilder createBuilder(RedisProperties.Pool pool) {
        if (isPoolEnabled(pool)) {
            return new LettuceConnectionConfiguration.PoolBuilderFactory().createBuilder(pool);
        }
        return LettuceClientConfiguration.builder();
    }

    private void applyProperties(
            LettuceClientConfiguration.LettuceClientConfigurationBuilder builder) {
        if (getProperties().isSsl()) {
            builder.useSsl();
        }
        if (getProperties().getTimeout() != null) {
            builder.commandTimeout(getProperties().getTimeout());
        }
        if (getProperties().getLettuce() != null) {
            RedisProperties.Lettuce lettuce = getProperties().getLettuce();
            if (lettuce.getShutdownTimeout() != null && !lettuce.getShutdownTimeout().isZero()) {
                builder.shutdownTimeout(getProperties().getLettuce().getShutdownTimeout());
            }
        }
        if (StringUtils.hasText(getProperties().getClientName())) {
            builder.clientName(getProperties().getClientName());
        }
    }

    private ClientOptions createClientOptions() {
        ClientOptions.Builder builder = initializeClientOptionsBuilder();
        Duration connectTimeout = getProperties().getConnectTimeout();
        if (connectTimeout != null) {
            builder.socketOptions(SocketOptions.builder().connectTimeout(connectTimeout).build());
        }
        return builder.timeoutOptions(TimeoutOptions.enabled()).build();
    }

    private ClientOptions.Builder initializeClientOptionsBuilder() {
        if (getProperties().getCluster() != null) {
            ClusterClientOptions.Builder builder = ClusterClientOptions.builder();
            RedisProperties.Lettuce.Cluster.Refresh refreshProperties = getProperties().getLettuce().getCluster().getRefresh();
            ClusterTopologyRefreshOptions.Builder refreshBuilder = ClusterTopologyRefreshOptions.builder()
                    .dynamicRefreshSources(refreshProperties.isDynamicRefreshSources());
            if (refreshProperties.getPeriod() != null) {
                refreshBuilder.enablePeriodicRefresh(refreshProperties.getPeriod());
            }
            if (refreshProperties.isAdaptive()) {
                refreshBuilder.enableAllAdaptiveRefreshTriggers();
            }
            return builder.topologyRefreshOptions(refreshBuilder.build());
        }
        return ClientOptions.builder();
    }

    private void customizeConfigurationFromUrl(LettuceClientConfiguration.LettuceClientConfigurationBuilder builder) {
        ConnectionInfo connectionInfo = parseUrl(getProperties().getUrl());
        if (connectionInfo.isUseSsl()) {
            builder.useSsl();
        }
    }

    /**
     * Inner class to allow optional commons-pool2 dependency.
     */
    private static class PoolBuilderFactory {

        LettuceClientConfiguration.LettuceClientConfigurationBuilder createBuilder(RedisProperties.Pool properties) {
            return LettucePoolingClientConfiguration.builder().poolConfig(getPoolConfig(properties));
        }

        private GenericObjectPoolConfig<?> getPoolConfig(RedisProperties.Pool properties) {
            GenericObjectPoolConfig<?> config = new GenericObjectPoolConfig<>();
            config.setMaxTotal(properties.getMaxActive());
            config.setMaxIdle(properties.getMaxIdle());
            config.setMinIdle(properties.getMinIdle());
            if (properties.getTimeBetweenEvictionRuns() != null) {
                config.setTimeBetweenEvictionRuns(properties.getTimeBetweenEvictionRuns());
            }
            if (properties.getMaxWait() != null) {
                config.setMaxWait(properties.getMaxWait());
            }
            return config;
        }

    }

}

到这里就可以直接通过配置 spring.redis.connect-type 来更改连接工厂了。保姆级服务贴个类目录图：
在这里插入图片描述

coding....

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
1
评论
解决 Kubernetes 部署 Redis Sentinel 本地无法连接的问题

解决 Kubernetes 部署 Redis Sentinel 本地无法连接的问题问题：在 kubernetes 部署 redis sentinel 时无法直接通过哨兵访问redis，因为哨兵返回的redis地址是pod ip或者是 service name，本地无法直接访问 pod ip，所以无法访问。解决方案：如果哨兵返回的是 service name可以直接配置本机 hosts文件，讲service name 路由至 redis master 节点的 nodePort, 这种可以
复制链接

扫一扫