redis版本5.0.5
测试redis Cluster主备切换、故障转移
-
下线一个从节点,此时它的主节点打印的日志
集群状态
-
下线一台主节点,此时它的从节点打印的日志
集群状态
测试主备切换时客户端状态
第一步:查看当前集群状态
可以看到六个节点都是可用状态,其中83.46的6379是81.64上的6380的从节点,计划Kill掉81.64上的6380主节点,然后观察83.46的6379节点日志
第二步:kill掉81.64上的6380
10:11:25:kill掉81.64上的6380,可以看到其从节点很快提示连接主节点失败,并且开始一秒钟一次的重连操作
此时查看集群的节点状态如下,可以看到槽 5461-10922在这个主节点上,此时整个reidis集群处于不可用状态
10:12:24:应用程序报错,redis操作超时
10:11:43 :在重连17次失败次数之后,从节点将主节点标记为失败,并且整个集群的状态切换为不可用,之后不甘心,又去尝试连接主节点
10:12:03:在重连20次失败后,从节点打印日志,等待投票选举,但是没有达到多数赞成,于是继续重连之前的主节点
10:12:14:提示选举失败,选举过期,又继续重连
10:12:45:选举成功胜出,成为了新的主节点,整个集群的状态变为可用
10:13:39:大概一分钟之后,redis客户端自动刷新了集群配置,成功连接上redis集群,此时主备切换和故障转移完成
此前项目中存在的问题
redis master宕机之后,会出现应用程序连接不上redis cluster的问题,需要重启服务才能解决
排查原因之后发现是spring boot 2.x版本默认使用了lettuce作为redis客户端,而lettuce默认是不开启自动刷新集群拓扑的,当redis master宕机并且集群完成故障转移/主从切换之后,客户端使用的还是之前错误的集群信息,就会导致应用程序一直连接不上redis集群。解决方案就是修改redis客户端配置,开启开启自适应刷新拓扑
配置文件如下
spring.redis.cluster.nodes=${redis.nodes}
spring.redis.password=${redis.pass}
spring.redis.timeout=60000
# 最大重定向次数
spring.redis.cluster.max-redirects=3
spring.redis.lettuce.pool.max-active=64
spring.redis.lettuce.pool.max-idle=16
spring.redis.lettuce.pool.min-idle=0
spring.redis.lettuce.pool.max-wait=60000ms
spring.redis.lettuce.shutdown-timeout=100ms
完整的配置类如下
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.lettuce.core.cluster.ClusterClientOptions;
import io.lettuce.core.cluster.ClusterTopologyRefreshOptions;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.data.redis.RedisProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisClusterConfiguration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.RedisNode;
import org.springframework.data.redis.connection.RedisPassword;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettucePoolingClientConfiguration;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import java.time.Duration;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
@Configuration
public class RedisConfig {
@Autowired
private RedisProperties redisProperties;
@Bean
@SuppressWarnings("all")
public RedisTemplate<String, Object> redisTemplate(RedisConnectionFactory factory) {
RedisTemplate<String, Object> template = new RedisTemplate<String, Object>();
template.setConnectionFactory(factory);
Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
ObjectMapper om = new ObjectMapper();
om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
jackson2JsonRedisSerializer.setObjectMapper(om);
StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
// key采用String的序列化方式
template.setKeySerializer(stringRedisSerializer);
// hash的key也采用String的序列化方式
template.setHashKeySerializer(stringRedisSerializer);
// value序列化方式采用jackson
template.setValueSerializer(jackson2JsonRedisSerializer);
// hash的value序列化方式采用jackson
template.setHashValueSerializer(jackson2JsonRedisSerializer);
template.afterPropertiesSet();
return template;
}
/**
* 为RedisTemplate配置Redis连接工厂实现
* LettuceConnectionFactory实现了RedisConnectionFactory接口
* 这里要注意的是,在构建LettuceConnectionFactory 时,如果不使用内置的destroyMethod,可能会导致Redis连接早于其它Bean被销毁
*
* @return 返回LettuceConnectionFactory
*/
@Bean(destroyMethod = "destroy")
public LettuceConnectionFactory lettuceConnectionFactory() {
List<String> clusterNodes = redisProperties.getCluster().getNodes();
Set<RedisNode> nodes = new HashSet<>();
clusterNodes.forEach(address -> nodes.add(new RedisNode(address.split(":")[0].trim(), Integer.parseInt(address.split(":")[1]))));
RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
clusterConfiguration.setClusterNodes(nodes);
clusterConfiguration.setPassword(RedisPassword.of(redisProperties.getPassword()));
clusterConfiguration.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());
GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
poolConfig.setMaxIdle(redisProperties.getLettuce().getPool().getMaxIdle());
poolConfig.setMinIdle(redisProperties.getLettuce().getPool().getMinIdle());
poolConfig.setMaxTotal(redisProperties.getLettuce().getPool().getMaxActive());
return new LettuceConnectionFactory(clusterConfiguration, getLettuceClientConfiguration(poolConfig));
}
/**
* 配置LettuceClientConfiguration 开启自适应刷新拓扑 包括线程池配置和安全项配置
*
* @param genericObjectPoolConfig common-pool2线程池
* @return lettuceClientConfiguration
*/
private LettuceClientConfiguration getLettuceClientConfiguration(GenericObjectPoolConfig genericObjectPoolConfig) {
/*
ClusterTopologyRefreshOptions配置用于开启自适应刷新和定时刷新。如自适应刷新不开启,Redis集群变更时将会导致连接异常!
*/
ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
//开启自适应刷新
//.enableAdaptiveRefreshTrigger(ClusterTopologyRefreshOptions.RefreshTrigger.MOVED_REDIRECT, ClusterTopologyRefreshOptions.RefreshTrigger.PERSISTENT_RECONNECTS)
//开启所有自适应刷新,MOVED,ASK,PERSISTENT都会触发
.enableAllAdaptiveRefreshTriggers()
// 自适应刷新超时时间(默认30秒)
.adaptiveRefreshTriggersTimeout(Duration.ofSeconds(25)) //默认关闭开启后时间为30秒
// 开周期刷新
.enablePeriodicRefresh(Duration.ofSeconds(20)) // 默认关闭开启后时间为60秒 ClusterTopologyRefreshOptions.DEFAULT_REFRESH_PERIOD 60 .enablePeriodicRefresh(Duration.ofSeconds(2)) = .enablePeriodicRefresh().refreshPeriod(Duration.ofSeconds(2))
.build();
return LettucePoolingClientConfiguration.builder()
.poolConfig(genericObjectPoolConfig)
.clientOptions(ClusterClientOptions.builder().topologyRefreshOptions(topologyRefreshOptions).build())
.build();
}
}