redis集群拓扑结构自动更新:使用Lettuce连接Cluster集群实例时异常处理

讓丄帝愛伱

已于 2023-05-02 18:33:59 修改

阅读量2.6k

点赞数 1

分类专栏： Linux 环境文章标签： linux redis

于 2023-04-22 17:06:24 首次发布

本文链接：https://blog.csdn.net/ximaiyao1984/article/details/130307210

版权

环境同时被 2 个专栏收录

218 篇文章 5 订阅

订阅专栏

Linux

168 篇文章 4 订阅

订阅专栏

问题：
使用lettuce连接Cluster集群实例，实例执行规格变更后，分片数有变化时，部分槽位（Slot）会迁移到新分片上，当客户端连接到新分片时会出现以下异常问题

java.lang.IllegalArgumentException: Connection to 100.123.70.194:6379 not allowed. This connection point is not known in the cluster view
exceptionStackTrace
io.lettuce.core.cluster.PooledClusterConnectionProvider.getConnectionAsync(PooledClusterConnectionProvider.java:359)
io.lettuce.core.cluster.ClusterDistributionChannelWriter.write(ClusterDistributionChannelWriter.java:93)
io.lettuce.core.cluster.ClusterCommand.complete(ClusterCommand.java:56)
io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:563)
io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:516)

org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out after 5 second(s)
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)

问题
redis连接后，隔一段时间，连接慢
Redis Cluster集群模式下master宕机或网络抖动等原因，主从切换期间报错：Redis command timed out等问题

Connection to X not allowed. This connection point is not known in the cluster view.
官网
 参考

解决方案

开启redis client的集群拓扑刷新功能，不同客户端，采用不同处理方式

jedis client默认自动支持（由于jedis通过自身异常反馈来识别重连、刷新服务端的集群信息机制，保证其自动故障恢复）
luttuce client 默认未开启，需要手动开启刷新
- springboot 1.x之前版本默认使用jedis，无需要手动开启刷新
- springboot 2.x，redis client默认为Lettuce，默认不支持拓扑刷新解决方案：
  - 使用jedis，不需要手动指定开启刷新
  - 使用lettuce，需要设置开启刷新节点拓扑策略
- springboot 2.3.0开始，支持集群拓扑刷新功能，属性配置开启即可

springboot 2.0.X

方法一

使用letture方式连接redis，需要设置开启刷新节点拓扑刷新策略

使用 RedisClusterClient.reloadPartitions 自动 reload pattitions

#redis cluster config
#RedisCluster集群节点及端口信息
spring.redis.cluster.nodes=192.168.50.29:6380,192.168.50.29:6381,192.168.50.29:6382,192.168.50.29:6383,192.168.50.29:6384,192.168.50.29:6385
#Redis密码
spring.redis.password=
#在群集中执行命令时要遵循的最大重定向数目
spring.redis.cluster.max-redirects=6
#Redis连接池在给定时间可以分配的最大连接数。使用负值无限制
spring.redis.jedis.pool.max-active=1000
#以毫秒为单位的连接超时时间
spring.redis.timeout=2000
#池中“空闲”连接的最大数量。使用负值表示无限数量的空闲连接
spring.redis.jedis.pool.max-idle=8
#目标为保持在池中的最小空闲连接数。这个设置只有在设置max-idle的情况下才有效果
spring.redis.jedis.pool.min-idle=5
#连接分配在池被耗尽时抛出异常之前应该阻塞的最长时间量（以毫秒为单位）。使用负值可以无限期地阻止
spring.redis.jedis.pool.max-wait=1000
#redis cluster只使用db0
spring.redis.index=0

import io.lettuce.core.ReadFrom;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.ClusterTopologyRefreshOptions;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
import org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration;
import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisStandaloneConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettucePoolingClientConfiguration;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;

import java.time.Duration;

/**
 * Redis配置类
 *
 * @author 007
 */
@Configuration
@AutoConfigureAfter(RedisAutoConfiguration.class)
public class RedisConfiguration {
    @Autowired
    private RedisProperties redisProperties;

    @Bean(destroyMethod = "destroy")
    public LettuceConnectionFactory redisConnectionFactory() {
        // redis单节点
        if (null == redisProperties.getCluster() || null == redisProperties.getCluster().getNodes()) {
            RedisStandaloneConfiguration configuration = new RedisStandaloneConfiguration(redisProperties.getHost(),
                    redisProperties.getPort());
            configuration.setPassword(redisProperties.getPassword());
            return new LettuceConnectionFactory(configuration);
        }

        // redis集群
        RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());
        redisClusterConfiguration.setPassword(redisProperties.getPassword());
        redisClusterConfiguration.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());

        GenericObjectPoolConfig genericObjectPoolConfig = new GenericObjectPoolConfig();
        genericObjectPoolConfig.setMaxTotal(redisProperties.getLettuce().getPool().getMaxActive());
        genericObjectPoolConfig.setMaxIdle(redisProperties.getLettuce().getPool().getMaxIdle());
        genericObjectPoolConfig.setMinIdle(redisProperties.getLettuce().getPool().getMinIdle());
        genericObjectPoolConfig.setMaxWaitMillis(redisProperties.getLettuce().getPool().getMaxWait().getSeconds());

        // 支持自适应集群拓扑刷新和动态刷新源
        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enableAllAdaptiveRefreshTriggers()
                // 开启自适应刷新
                .enableAdaptiveRefreshTrigger()
                // 开启定时刷新
                .enablePeriodicRefresh(Duration.ofSeconds(5))
                .build();

        ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
                .topologyRefreshOptions(clusterTopologyRefreshOptions).build();
			/*
			 ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
			                                 .topologyRefreshOptions(clusterTopologyRefreshOptions)//拓扑刷新
			                                 .disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
			                                 .autoReconnect(true)
			                                 .socketOptions(SocketOptions.builder().keepAlive(true).build())
			　　　　　　　　　　　　　　　　       .validateClusterNodeMembership(false)// 取消校验集群节点的成员关系
			                                 .build();
			*/

        LettuceClientConfiguration lettuceClientConfiguration = LettucePoolingClientConfiguration.builder()
                .poolConfig(genericObjectPoolConfig)
                .readFrom(ReadFrom.SLAVE_PREFERRED) // 使用ReadFrom.SLAVE_PREFERRED读写分离
                .clientOptions(clusterClientOptions).build();

        LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(redisClusterConfiguration, lettuceClientConfiguration);
        lettuceConnectionFactory.setShareNativeConnection(false);// 是否允许多个线程操作共用同一个缓存连接，默认 true，false 时每个操作都将开辟新的连接
        lettuceConnectionFactory.resetConnection();// 重置底层共享连接, 在接下来的访问时初始化
        return lettuceConnectionFactory;
    }

 	//自定义一个RedisTemplate
    @Bean
    @SuppressWarnings("all")
    public RedisTemplate<String, Object> redisTemplate(@Qualifier("lettuceConnectionFactoryUvPv") LettuceConnectionFactory factory) {
        RedisTemplate<String, Object> template = new RedisTemplate<>();
        // 一定要注入RedisTemplate和redisTemplate.setConnectionFactory(lettuceConnectionFactory);
        template.setConnectionFactory(factory);
 
        //Json序列化配置
        Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer<Object>(Object.class);
        ObjectMapper om = new ObjectMapper();
        om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
        om.activateDefaultTyping(om.getPolymorphicTypeValidator());
        om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
        //解决序列化问题
        om.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        jackson2JsonRedisSerializer.setObjectMapper(om);
 
        //String的序列化
        StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
 
        //key采用String的序列化方式
        template.setKeySerializer(stringRedisSerializer);
        //hash的key也采用String的序列化方式
        template.setHashKeySerializer(stringRedisSerializer);
 
        //value序列化方式采用jackson
        template.setValueSerializer(jackson2JsonRedisSerializer);
 
        //hash的value序列化方式采用jackson
        template.setHashValueSerializer(jackson2JsonRedisSerializer);
        template.afterPropertiesSet();
 
        return template;
    }
}

import io.lettuce.core.ClientOptions;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.ClusterTopologyRefreshOptions;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisNode;
import org.springframework.data.redis.connection.RedisPassword;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettucePoolingClientConfiguration;
import org.springframework.stereotype.Component;

import java.time.Duration;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

/**
 * @author : 007
 * @version : 2.0
 * @date : 2020/7/27 10:19
 */
@Component
public class RedisConfig {

    @Autowired
    private RedisProperties redisProperties;

    public GenericObjectPoolConfig<?> genericObjectPoolConfig(RedisProperties.Pool properties) {
        GenericObjectPoolConfig<?> config = new GenericObjectPoolConfig<>();
        config.setMaxTotal(properties.getMaxActive());
        config.setMaxIdle(properties.getMaxIdle());
        config.setMinIdle(properties.getMinIdle());
        if (properties.getTimeBetweenEvictionRuns() != null) {
            config.setTimeBetweenEvictionRunsMillis(properties.getTimeBetweenEvictionRuns().toMillis());
        }
        if (properties.getMaxWait() != null) {
            config.setMaxWaitMillis(properties.getMaxWait().toMillis());
        }
        return config;
    }

    @Bean(destroyMethod = "destroy")
    public LettuceConnectionFactory lettuceConnectionFactory() {

        //开启 自适应集群拓扑刷新和周期拓扑刷新
        ClusterTopologyRefreshOptions clusterTopologyRefreshOptions =  ClusterTopologyRefreshOptions.builder()
                // 开启全部自适应刷新
                .enableAllAdaptiveRefreshTriggers() // 开启自适应刷新,自适应刷新不开启,Redis集群变更时将会导致连接异常
                // 开启自适应刷新
                //.enableAdaptiveRefreshTrigger()
                // 自适应刷新超时时间(默认30秒)
                .adaptiveRefreshTriggersTimeout(Duration.ofSeconds(30)) //默认关闭开启后时间为30秒
                // 开周期刷新
                .enablePeriodicRefresh(Duration.ofSeconds(20))  // 默认关闭开启后时间为60秒 ClusterTopologyRefreshOptions.DEFAULT_REFRESH_PERIOD 60  .enablePeriodicRefresh(Duration.ofSeconds(2)) = .enablePeriodicRefresh().refreshPeriod(Duration.ofSeconds(2))
                .build();

        // https://github.com/lettuce-io/lettuce-core/wiki/Client-Options
        ClientOptions clientOptions = ClusterClientOptions.builder()
                .topologyRefreshOptions(clusterTopologyRefreshOptions)
                .build();

        LettuceClientConfiguration clientConfig = LettucePoolingClientConfiguration.builder()
                .poolConfig(genericObjectPoolConfig(redisProperties.getJedis().getPool()))
                //.readFrom(ReadFrom.MASTER_PREFERRED)
                .clientOptions(clientOptions)
                .commandTimeout(redisProperties.getTimeout()) //默认RedisURI.DEFAULT_TIMEOUT 60
                .build();

        List<String> clusterNodes = redisProperties.getCluster().getNodes();
        Set<RedisNode> nodes = new HashSet<RedisNode>();
        clusterNodes.forEach(address -> nodes.add(new RedisNode(address.split(":")[0].trim(), Integer.valueOf(address.split(":")[1]))));

        RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
        clusterConfiguration.setClusterNodes(nodes);
        clusterConfiguration.setPassword(RedisPassword.of(redisProperties.getPassword()));
        clusterConfiguration.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());

        LettuceConnectionFactory lettuceConnectionFactory = new LettuceConnectionFactory(clusterConfiguration, clientConfig);
        // lettuceConnectionFactory.setShareNativeConnection(false); //是否允许多个线程操作共用同一个缓存连接，默认true，false时每个操作都将开辟新的连接
        // lettuceConnectionFactory.resetConnection(); // 重置底层共享连接, 在接下来的访问时初始化
        return lettuceConnectionFactory;
    }
}

public LettuceConnectionFactory redisConnectionFactory(ClientResources clientResources) {
    // redis单节点
    if (null == redisProperties.getCluster() || null == redisProperties.getCluster().getNodes()) {
        RedisStandaloneConfiguration configuration = new RedisStandaloneConfiguration(redisProperties.getHost(),
                redisProperties.getPort());
        configuration.setPassword(redisProperties.getPassword());
        return new LettuceConnectionFactory(configuration);
    }

    ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
            // //按照周期刷新拓扑
            .enablePeriodicRefresh(Duration.ofSeconds(30))
            //根据事件刷新拓扑
            .enableAllAdaptiveRefreshTriggers()
            .build();

    ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
            //redis命令超时时间,超时后才会使用新的拓扑信息重新建立连接
            .timeoutOptions(TimeoutOptions.enabled(Duration.ofSeconds(10)))
            .topologyRefreshOptions(topologyRefreshOptions)
            .build();

    LettuceClientConfiguration clientConfiguration = LettuceClientConfiguration.builder()
            .clientResources(clientResources)
            .clientOptions(clusterClientOptions)
            .build();

    RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());
    if (redisProperties.getCluster().getMaxRedirects() != null) {
        clusterConfig.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());
    }
    clusterConfig.setPassword(RedisPassword.of(redisProperties.getPassword()));

    return new LettuceConnectionFactory(clusterConfig, clientConfiguration);
}

方法二

改用jedis方式连接redis，使用jedis客户端的服务可以在主从切换后15秒恢复

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
    <exclusions>
        <exclusion>
            <groupId>io.lettuce</groupId>
            <artifactId>lettuce-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
</dependency>

注意配置文件把lettuce.pool换成jedis.pool

lettuce 配置信息

spring.redis.lettuce.pool.max-active=10
spring.redis.lettuce.pool.max-idle=10
spring.redis.lettuce.pool.max-wait=10
spring.redis.lettuce.pool.min-idle=5
spring.redis.lettuce.shutdown-timeout=1S

jedis 配置信息

spring.redis.jedis.pool.max-active=10
spring.redis.jedis.pool.max-idle=10
spring.redis.jedis.pool.max-wait=10
spring.redis.jedis.pool.min-idle=5

spring:
  redis:
    password: xxx
    host: 172.16.0.x
    port: 6579
    timeout: 5000
    jedis:
      pool:
              #最大连接数据库连接数,设 0 为没有限制
        max-active: 8
              #最大等待连接中的数量,设 0 为没有限制
        max-idle: 8
              #最大建立连接等待时间。如果超过此时间将接到异常。设为-1表示无限制。
        max-wait: -1ms
              #最小等待连接中的数量,设 0 为没有限制
        min-idle: 0
    #lettuce:
      #pool:
        #max-active: ${redis.config.maxTotal:1024}
        #max-idle: ${redis.config.maxIdle:50}
        #min-idle: ${redis.config.minIdle:1}
        #max-wait: ${redis.config.maxWaitMillis:5000}

springboot 2.3

配置属性

spring.redis.lettuce.cluster.refresh.adaptive= true
spring.redis.lettuce.cluster.refresh.period=30000

spring:
  redis:
	lettuce:
	  cluster:
		refresh:
		  adaptive: true
		  period: 30000    # 30秒自动刷新一次

讓丄帝愛伱

关注

1
点赞
踩
10

收藏

觉得还不错? 一键收藏
打赏
0
评论
redis集群拓扑结构自动更新:使用Lettuce连接Cluster集群实例时异常处理

错误：依赖检测失败： liberation-fonts 被 google-chrome-stable-72.0.3626.109-1.x86_64 需要。
复制链接

扫一扫