简介
集群是一个比较模糊的概念,只要大于一台机器共享(只读或可读可写)了数据的模式,就可以说是集群.
因此,redis的集群模式可以是:
- 主从复制模式
- sentinel模式
- cluster模式
这里说的集群指的是cluster模式
,它兼具主从复制
的数据共享和sentinel
的自动故障转移功能.
搭建步骤
一般分为四步:
- 修改配置文件,开启集群功能
- 节点互通,通过
meet
命令让各个节点相互认识 - 分配槽点
slot
,只需要在已经规划的主节点上分配,各个主节点的slot的合集保证在0-16383
范围中 - 分片,简单理解成主从复制里面设置从节点的过程
实现方式
配置开启
配置文件还是要修改的(手动狗头),核心的几个参考如下:
# 开启集群模式,默认关闭
cluster-enabled yes
# 指定配置文件,只需要在这里指出即可,无需手动创建
cluster-config-file nodes-6379.conf
cluster-node-timeout 15000
# 当一个节点挂掉,并且其他节点没有进行故障转移的时候集群不可用.默认yes,一般设置为no
cluster-require-full-coverage no
cluster-replica-no-failover no
其他步骤
上面所说的步骤,大致相当于思路,而实现的方式则不止一种,大概分为
- 原生搭建
官方(曾经)推荐的ruby脚本搭建1- 最新版本
5.0.7
提供的redis-cli
架构
原生
-
meet
redis-cli -h ${host} -p ${port} CLUSTER meet ${another_host} ${another_port}
这里保证所有节点是互通的就行.
-
指派槽(slot)
一个集群,总共有16384(0~16383)个槽,这些槽口是给即将分配为主节点角色准备的,这里采用平分的方式,参看上图所示的架构图.
redis-cli -h ${host} -p ${port} CLUSTER addslots slot [slot...]
-
分片(replication)
说白了就是把自己设置为谁的从节点,涉及命令结构如下:
# 表示port是NODE_ID为${anoter_node_id}的节点的从节点 redis-cli -h ${host} -p ${port} CLUSTER replicate ${anoter_node_id}
其中node_id 通过
redis-cli cluster nodes
查看,比如我的输出如下:127.0.0.1:7000> cluster nodes 0cf6b1eed9b0e8170d6173cdd66534a451892785 127.0.0.1:7001@17001 master - 0 1579248861000 1 connected 5462-10923 3eb3899fc132536a25588f92644f7c48fd006567 127.0.0.1:7007@17007 master - 0 1579248861042 8 connected 9d9cdf34f4a13601e62746795a81759bdca35fa5 127.0.0.1:7005@17005 master - 0 1579248860000 5 connected c608cc216dea7ea52bb26b0c4a263d16043bbfdc 127.0.0.1:7003@17003 master - 0 1579248859000 3 connected 027171c3d046ce423a7d4d493e93da4b59df844e 127.0.0.1:7006@17006 master - 0 1579248860000 6 connected fdb567abbf5f06e7be18397f787db1c4a110b62f 127.0.0.1:7000@17000 myself,master - 0 1579248859000 0 connected 0-5461 14da1ca1e7586e6ee61e0af530f4da7c1a166eae 127.0.0.1:7004@17004 master - 0 1579248862042 4 connected a50298a2ff7460acd869b49863f9e45b70468af2 127.0.0.1:7008@17008 master - 0 1579248863045 7 connected c4d7ace7a77994ab634f1e71029581b5b1137b5d 127.0.0.1:7002@17002 master - 0 1579248860000 2 connected 10924-16383
那我应该根据分片命令
redis-cli -h 127.0.0.1 -p 7003 CLUSTER replicate fdb567abbf5f06e7be18397f787db1c4a110b62f redis-cli -h 127.0.0.1 -p 7004 CLUSTER replicate fdb567abbf5f06e7be18397f787db1c4a110b62f redis-cli -h 127.0.0.1 -p 7005 CLUSTER replicate 0cf6b1eed9b0e8170d6173cdd66534a451892785 redis-cli -h 127.0.0.1 -p 7006 CLUSTER replicate 0cf6b1eed9b0e8170d6173cdd66534a451892785 redis-cli -h 127.0.0.1 -p 7007 CLUSTER replicate c4d7ace7a77994ab634f1e71029581b5b1137b5d redis-cli -h 127.0.0.1 -p 7008 CLUSTER replicate c4d7ace7a77994ab634f1e71029581b5b1137b5d
-
测试
yan@yan-PC:/etc/redis$ redis-cli -p 7003 -c 127.0.0.1:7003> get hello -> Redirected to slot [866] located at 127.0.0.1:7000 "www" 127.0.0.1:7000> exit yan@yan-PC:/etc/redis$ redis-cli -p 7008 -c 127.0.0.1:7008> get hello -> Redirected to slot [866] located at 127.0.0.1:7000 "www"
ruby方式
首先安装Ruby,
可以参考ruby安装及gem源配置
$REDIS_HOME/src/redis-trib.rb create --replicas 2 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 127.0.0.1:7007 127.0.0.1:7008
我的版本是不给用的,直接报错.不过低版本的可以试试.
注意
这个方式已被官方废弃,当使用redis-trib.rb
时会报错并给出最新提示:
WARNING: redis-trib.rb is not longer available!
You should use redis-cli instead.
All commands and features belonging to redis-trib.rb have been moved
to redis-cli.
In order to use them you should call redis-cli with the --cluster
option followed by the subcommand name, arguments and options.
Use the following syntax:
redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
Example:
redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 127.0.0.1:7007 127.0.0.1:7008 --cluster-replicas 2
To get help about all subcommands, type:
redis-cli --cluster help
redis-cli
上边用原生方式搭建过集群了,这里先把所有开启的redis集群服务解散,包括清空数据,下线节点,清空槽点三步.
for port in $*
do
# 清空数据
redis-cli -p ${port} FLUSHALL
# 下线当前节点
redis-cli -p ${port} CLUSTER RESET hard
# 删除槽点
redis-cli -p ${port} CLUSTER FLUSHSLOTS
done
调用重置脚本
/etc/redis/scripts/forget-nodes.sh 7000 7001 7002 7003 7004 7005 7006 7007 7008
搭建集群
一条命令搞定,和上面提到的用Ruby
的方式非常相似:
redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 127.0.0.1:7007 127.0.0.1:7008 --cluster-replicas 2
这条命令包含了meet
,addslots
和分片
三个操作.
>>> Performing hash slots allocation on 9 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
Adding replica 127.0.0.1:7005 to 127.0.0.1:7000
Adding replica 127.0.0.1:7006 to 127.0.0.1:7001
Adding replica 127.0.0.1:7007 to 127.0.0.1:7001
Adding replica 127.0.0.1:7008 to 127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 4201cac377d8841cd80c90ce77763ecbcddae2b6 127.0.0.1:7000
slots:[0-5460] (5461 slots) master
M: b250338f2f3b37e17e37bedb7377be2df9a525e9 127.0.0.1:7001
slots:[5461-10922] (5462 slots) master
M: 951bda2d04f7064576c12609585de60900b55e36 127.0.0.1:7002
slots:[10923-16383] (5461 slots) master
S: 960311a146fad7b9ef5e30457140fe20a078d3b1 127.0.0.1:7003
replicates 951bda2d04f7064576c12609585de60900b55e36
S: 67400048cbc03528899528e532a8126679dc8f30 127.0.0.1:7004
replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
S: 4e3fc23f76ab8a9ff5896cbfa9a9086bda0de794 127.0.0.1:7005
replicates 951bda2d04f7064576c12609585de60900b55e36
S: 9f5a40ea737ec7d1cfe7d9de1ddec8f3215f3d70 127.0.0.1:7006
replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
S: 928f07fc036ee64ab26a26a380dd20daaecb7de0 127.0.0.1:7007
replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
S: e558b0af32c524ac41ec89de6de8d1bd6907fd78 127.0.0.1:7008
replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..........
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 4201cac377d8841cd80c90ce77763ecbcddae2b6 127.0.0.1:7000
slots:[0-5460] (5461 slots) master
2 additional replica(s)
S: 960311a146fad7b9ef5e30457140fe20a078d3b1 127.0.0.1:7003
slots: (0 slots) slave
replicates 951bda2d04f7064576c12609585de60900b55e36
S: 4e3fc23f76ab8a9ff5896cbfa9a9086bda0de794 127.0.0.1:7005
slots: (0 slots) slave
replicates 951bda2d04f7064576c12609585de60900b55e36
S: 67400048cbc03528899528e532a8126679dc8f30 127.0.0.1:7004
slots: (0 slots) slave
replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
M: b250338f2f3b37e17e37bedb7377be2df9a525e9 127.0.0.1:7001
slots:[5461-10922] (5462 slots) master
2 additional replica(s)
S: 9f5a40ea737ec7d1cfe7d9de1ddec8f3215f3d70 127.0.0.1:7006
slots: (0 slots) slave
replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
S: 928f07fc036ee64ab26a26a380dd20daaecb7de0 127.0.0.1:7007
slots: (0 slots) slave
replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
S: e558b0af32c524ac41ec89de6de8d1bd6907fd78 127.0.0.1:7008
slots: (0 slots) slave
replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
M: 951bda2d04f7064576c12609585de60900b55e36 127.0.0.1:7002
slots:[10923-16383] (5461 slots) master
2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Java集成
Jedis
HashSet<HostAndPort> hostAndPorts = new HashSet<HostAndPort>() {{
add(new HostAndPort("127.0.0.1", 7000));
add(new HostAndPort("127.0.0.1", 7001));
add(new HostAndPort("127.0.0.1", 7002));
add(new HostAndPort("127.0.0.1", 7003));
add(new HostAndPort("127.0.0.1", 7004));
add(new HostAndPort("127.0.0.1", 7005));
add(new HostAndPort("127.0.0.1", 7006));
add(new HostAndPort("127.0.0.1", 7007));
add(new HostAndPort("127.0.0.1", 7008));
}};
JedisCluster jedisCluster = new JedisCluster(hostAndPorts);
jedisCluster.set("datetime", LocalDateTime.now().toString());
System.out.println(jedisCluster.get("datetime"));
Springboot-redis
yml
这里的连接池随便配的,生产环境根据自己业务量决定,原则参考资源池(commons-pool)参数
spring:
redis:
database: 0
host: ${REDIS_HOST:127.0.0.1}
password: ${REDIS_PASSWORD:}
cluster:
nodes: 127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002,127.0.0.1:7003,127.0.0.1:7004,127.0.0.1:7005,127.0.0.1:7006,127.0.0.1:7007,127.0.0.1:7008
max-redirects: 5
jedis:
pool:
max-active: -1
max-idle: 0
max-wait: -1
min-idle: 0
port: 6379
timeout: 0
RedisConfiguration
这里只列举与Cluster
模式相关的配置,其他序列化配置参考spring boot集成redis的基本配置
@Autowired private RedisProperties redisProperties;
@Bean
@Primary
public JedisConnectionFactory jedisConnectionFactory(JedisClientConfiguration jedisClientConfiguration) {
RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
List<String> nodes = redisProperties.getCluster().getNodes();
Set<RedisNode> redisNodes = nodes.stream().map(n -> {
String[] split = n.split(":");
String host = split[0];
String port = split[1];
return new RedisNode(host, Integer.parseInt(port));
}).collect(Collectors.toSet());
clusterConfiguration.setClusterNodes(redisNodes);
//(单机|哨兵|集群)配置 + 客户端配置 = jedis连接工厂
return new JedisConnectionFactory(clusterConfiguration, jedisClientConfiguration);
}
@Bean
@Primary
public JedisPoolConfig jedisPoolConfig() {
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
RedisProperties.Pool pool = redisProperties.getJedis().getPool();
jedisPoolConfig.setMaxIdle(pool.getMaxIdle());
jedisPoolConfig.setMaxTotal(pool.getMaxIdle());
return jedisPoolConfig;
}
@Bean
@Primary
public JedisClientConfiguration jedisClientConfiguration(JedisPoolConfig jedisPoolConfig) {
JedisClientConfiguration.JedisPoolingClientConfigurationBuilder jpcb =
(JedisClientConfiguration.JedisPoolingClientConfigurationBuilder) JedisClientConfiguration.builder();
jpcb.poolConfig(jedisPoolConfig);
JedisClientConfiguration.JedisClientConfigurationBuilder jedisClientConfigurationBuilder = JedisClientConfiguration.builder();
jedisClientConfigurationBuilder.connectTimeout(redisProperties.getTimeout());// connection timeout
return jpcb.build();
}
@Bean
@Scope(scopeName = "prototype")
public RedisConnection redisConnection(RedisConnectionFactory factory) {
return factory.getConnection();
}
注意
Springboot提供的配置属性,与Jedis要求的不是很一致,需要自己配置,比如最大连接数,Jedis需要maxTotal
,而Springboot中没有提供,所以需要在RedisConfiguration
中自己根据实际情况配置.
RedisApplication
@SpringBootApplication
public class RedisApplication implements CommandLineRunner {
@Autowired
private RedisConnection redisConnection;
public static void main(String[] args) {
SpringApplication.run(RedisApplication.class, args);
}
@SuppressWarnings("all")
public void run(String... strings) {
redisConnection.set("datetime".getBytes(), LocalDateTime.now().toString().getBytes());
String s = new String(redisConnection.get("datetime".getBytes()), StandardCharsets.UTF_8);
System.out.println(s);
}
}
总结
常用命令
meet
redis-cli -h ${host} -p ${port} CLUSTER meet ${another_host} ${another_port}
删除节点
redis-cli -h ${host} -p ${port} CLUSTER FORGET ${NODE_ID}
新版(5.0.7
)redis-cli
提供了很多关于集群的操作,可以通过CLUSTER HELP
查看.
现在如果再使用
redis-trib.rb
来搭建会直接报错并建议使用redis-cli
命令. ↩︎