Redis学习笔记(八):集群

简介

集群是一个比较模糊的概念,只要大于一台机器共享(只读或可读可写)了数据的模式,就可以说是集群.

因此,redis的集群模式可以是:

1. 主从复制模式
2. sentinel模式
3. cluster模式

这里说的集群指的是cluster模式,它兼具主从复制的数据共享和sentinel的自动故障转移功能.

搭建步骤

一般分为四步:

1. 修改配置文件,开启集群功能
2. 节点互通,通过meet命令让各个节点相互认识
3. 分配槽点slot,只需要在已经规划的主节点上分配,各个主节点的slot的合集保证在0-16383范围中
4. 分片,简单理解成主从复制里面设置从节点的过程

实现方式

配置开启

配置文件还是要修改的(手动狗头),核心的几个参考如下:

# 开启集群模式,默认关闭
cluster-enabled yes
# 指定配置文件,只需要在这里指出即可,无需手动创建
cluster-config-file nodes-6379.conf
cluster-node-timeout 15000
# 当一个节点挂掉,并且其他节点没有进行故障转移的时候集群不可用.默认yes,一般设置为no
cluster-require-full-coverage no
cluster-replica-no-failover no

其他步骤

上面所说的步骤,大致相当于思路,而实现的方式则不止一种,大概分为

1. 原生搭建
2. 官方(曾经)推荐的ruby脚本搭建¹
3. 最新版本5.0.7提供的redis-cli

架构

原生

1. meet

   redis-cli -h ${host} -p ${port} CLUSTER meet ${another_host} ${another_port}
   

   这里保证所有节点是互通的就行.

2. 指派槽(slot)

   一个集群,总共有16384(0~16383)个槽,这些槽口是给即将分配为主节点角色准备的,这里采用平分的方式,参看上图所示的架构图.

   redis-cli -h ${host} -p ${port} CLUSTER addslots slot [slot...]
   

3. 分片(replication)

   说白了就是把自己设置为谁的从节点,涉及命令结构如下:

   # 表示port是NODE_ID为${anoter_node_id}的节点的从节点
   redis-cli -h ${host} -p ${port} CLUSTER replicate ${anoter_node_id}
   

   其中node_id 通过redis-cli cluster nodes查看,比如我的输出如下:

   127.0.0.1:7000> cluster nodes
   0cf6b1eed9b0e8170d6173cdd66534a451892785 127.0.0.1:7001@17001 master - 0 1579248861000 1 connected 5462-10923
   3eb3899fc132536a25588f92644f7c48fd006567 127.0.0.1:7007@17007 master - 0 1579248861042 8 connected
   9d9cdf34f4a13601e62746795a81759bdca35fa5 127.0.0.1:7005@17005 master - 0 1579248860000 5 connected
   c608cc216dea7ea52bb26b0c4a263d16043bbfdc 127.0.0.1:7003@17003 master - 0 1579248859000 3 connected
   027171c3d046ce423a7d4d493e93da4b59df844e 127.0.0.1:7006@17006 master - 0 1579248860000 6 connected
   fdb567abbf5f06e7be18397f787db1c4a110b62f 127.0.0.1:7000@17000 myself,master - 0 1579248859000 0 connected 0-5461
   14da1ca1e7586e6ee61e0af530f4da7c1a166eae 127.0.0.1:7004@17004 master - 0 1579248862042 4 connected
   a50298a2ff7460acd869b49863f9e45b70468af2 127.0.0.1:7008@17008 master - 0 1579248863045 7 connected
   c4d7ace7a77994ab634f1e71029581b5b1137b5d 127.0.0.1:7002@17002 master - 0 1579248860000 2 connected 10924-16383
   

   那我应该根据分片命令

   redis-cli -h 127.0.0.1 -p 7003 CLUSTER replicate fdb567abbf5f06e7be18397f787db1c4a110b62f
   redis-cli -h 127.0.0.1 -p 7004 CLUSTER replicate fdb567abbf5f06e7be18397f787db1c4a110b62f
   redis-cli -h 127.0.0.1 -p 7005 CLUSTER replicate 0cf6b1eed9b0e8170d6173cdd66534a451892785
   redis-cli -h 127.0.0.1 -p 7006 CLUSTER replicate 0cf6b1eed9b0e8170d6173cdd66534a451892785
   redis-cli -h 127.0.0.1 -p 7007 CLUSTER replicate c4d7ace7a77994ab634f1e71029581b5b1137b5d
   redis-cli -h 127.0.0.1 -p 7008 CLUSTER replicate c4d7ace7a77994ab634f1e71029581b5b1137b5d
   

4. 测试

   yan@yan-PC:/etc/redis$ redis-cli -p 7003 -c
   127.0.0.1:7003> get hello
   -> Redirected to slot [866] located at 127.0.0.1:7000
   "www"
   127.0.0.1:7000> exit
   yan@yan-PC:/etc/redis$ redis-cli -p 7008 -c
   127.0.0.1:7008> get hello
   -> Redirected to slot [866] located at 127.0.0.1:7000
   "www"
   

ruby方式

首先安装Ruby,可以参考ruby安装及gem源配置

$REDIS_HOME/src/redis-trib.rb create --replicas 2 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 127.0.0.1:7007 127.0.0.1:7008

我的版本是不给用的,直接报错.不过低版本的可以试试.

注意

这个方式已被官方废弃,当使用redis-trib.rb时会报错并给出最新提示:

WARNING: redis-trib.rb is not longer available!
You should use redis-cli instead.

All commands and features belonging to redis-trib.rb have been moved
to redis-cli.
In order to use them you should call redis-cli with the --cluster
option followed by the subcommand name, arguments and options.

Use the following syntax:
redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]

Example:
redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 127.0.0.1:7007 127.0.0.1:7008 --cluster-replicas 2

To get help about all subcommands, type:
redis-cli --cluster help

redis-cli

上边用原生方式搭建过集群了,这里先把所有开启的redis集群服务解散,包括清空数据,下线节点,清空槽点三步.

for port in $*
do
	# 清空数据
	redis-cli -p ${port} FLUSHALL
	# 下线当前节点
	redis-cli -p ${port} CLUSTER RESET hard
	# 删除槽点
	redis-cli -p ${port} CLUSTER FLUSHSLOTS
done

调用重置脚本

/etc/redis/scripts/forget-nodes.sh 7000 7001 7002 7003 7004 7005 7006 7007 7008

搭建集群

一条命令搞定,和上面提到的用Ruby的方式非常相似:

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 127.0.0.1:7007 127.0.0.1:7008 --cluster-replicas 2

这条命令包含了meet,addslots和分片三个操作.

>>> Performing hash slots allocation on 9 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
Adding replica 127.0.0.1:7005 to 127.0.0.1:7000
Adding replica 127.0.0.1:7006 to 127.0.0.1:7001
Adding replica 127.0.0.1:7007 to 127.0.0.1:7001
Adding replica 127.0.0.1:7008 to 127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 4201cac377d8841cd80c90ce77763ecbcddae2b6 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
M: b250338f2f3b37e17e37bedb7377be2df9a525e9 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
M: 951bda2d04f7064576c12609585de60900b55e36 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
S: 960311a146fad7b9ef5e30457140fe20a078d3b1 127.0.0.1:7003
   replicates 951bda2d04f7064576c12609585de60900b55e36
S: 67400048cbc03528899528e532a8126679dc8f30 127.0.0.1:7004
   replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
S: 4e3fc23f76ab8a9ff5896cbfa9a9086bda0de794 127.0.0.1:7005
   replicates 951bda2d04f7064576c12609585de60900b55e36
S: 9f5a40ea737ec7d1cfe7d9de1ddec8f3215f3d70 127.0.0.1:7006
   replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
S: 928f07fc036ee64ab26a26a380dd20daaecb7de0 127.0.0.1:7007
   replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
S: e558b0af32c524ac41ec89de6de8d1bd6907fd78 127.0.0.1:7008
   replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..........
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 4201cac377d8841cd80c90ce77763ecbcddae2b6 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
   2 additional replica(s)
S: 960311a146fad7b9ef5e30457140fe20a078d3b1 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 951bda2d04f7064576c12609585de60900b55e36
S: 4e3fc23f76ab8a9ff5896cbfa9a9086bda0de794 127.0.0.1:7005
   slots: (0 slots) slave
   replicates 951bda2d04f7064576c12609585de60900b55e36
S: 67400048cbc03528899528e532a8126679dc8f30 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
M: b250338f2f3b37e17e37bedb7377be2df9a525e9 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   2 additional replica(s)
S: 9f5a40ea737ec7d1cfe7d9de1ddec8f3215f3d70 127.0.0.1:7006
   slots: (0 slots) slave
   replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
S: 928f07fc036ee64ab26a26a380dd20daaecb7de0 127.0.0.1:7007
   slots: (0 slots) slave
   replicates 4201cac377d8841cd80c90ce77763ecbcddae2b6
S: e558b0af32c524ac41ec89de6de8d1bd6907fd78 127.0.0.1:7008
   slots: (0 slots) slave
   replicates b250338f2f3b37e17e37bedb7377be2df9a525e9
M: 951bda2d04f7064576c12609585de60900b55e36 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
   2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Java集成

Jedis

        HashSet<HostAndPort> hostAndPorts = new HashSet<HostAndPort>() {{
            add(new HostAndPort("127.0.0.1", 7000));
            add(new HostAndPort("127.0.0.1", 7001));
            add(new HostAndPort("127.0.0.1", 7002));
            add(new HostAndPort("127.0.0.1", 7003));
            add(new HostAndPort("127.0.0.1", 7004));
            add(new HostAndPort("127.0.0.1", 7005));
            add(new HostAndPort("127.0.0.1", 7006));
            add(new HostAndPort("127.0.0.1", 7007));
            add(new HostAndPort("127.0.0.1", 7008));
        }};
        JedisCluster jedisCluster = new JedisCluster(hostAndPorts);
        jedisCluster.set("datetime", LocalDateTime.now().toString());
        System.out.println(jedisCluster.get("datetime"));

Springboot-redis

yml

这里的连接池随便配的,生产环境根据自己业务量决定,原则参考资源池（commons-pool）参数

spring:
  redis:
    database: 0
    host: ${REDIS_HOST:127.0.0.1}
    password: ${REDIS_PASSWORD:}
    cluster:
      nodes: 127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002,127.0.0.1:7003,127.0.0.1:7004,127.0.0.1:7005,127.0.0.1:7006,127.0.0.1:7007,127.0.0.1:7008
      max-redirects: 5
    jedis:
      pool:
        max-active: -1
        max-idle: 0
        max-wait: -1
        min-idle: 0
    port: 6379
    timeout: 0

RedisConfiguration

这里只列举与Cluster模式相关的配置,其他序列化配置参考spring boot集成redis的基本配置

	@Autowired private RedisProperties redisProperties;

    @Bean
    @Primary
    public JedisConnectionFactory jedisConnectionFactory(JedisClientConfiguration jedisClientConfiguration) {
        RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
        List<String> nodes = redisProperties.getCluster().getNodes();
        Set<RedisNode> redisNodes = nodes.stream().map(n -> {
            String[] split = n.split(":");
            String host = split[0];
            String port = split[1];
            return new RedisNode(host, Integer.parseInt(port));
        }).collect(Collectors.toSet());
        clusterConfiguration.setClusterNodes(redisNodes);
        //(单机|哨兵|集群)配置 + 客户端配置 = jedis连接工厂
        return new JedisConnectionFactory(clusterConfiguration, jedisClientConfiguration);
    }

    @Bean
    @Primary
    public JedisPoolConfig jedisPoolConfig() {
        JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
        RedisProperties.Pool pool = redisProperties.getJedis().getPool();
        jedisPoolConfig.setMaxIdle(pool.getMaxIdle());
        jedisPoolConfig.setMaxTotal(pool.getMaxIdle());
        return jedisPoolConfig;
    }

    @Bean
    @Primary
    public JedisClientConfiguration jedisClientConfiguration(JedisPoolConfig jedisPoolConfig) {
        JedisClientConfiguration.JedisPoolingClientConfigurationBuilder jpcb =
                (JedisClientConfiguration.JedisPoolingClientConfigurationBuilder) JedisClientConfiguration.builder();
        jpcb.poolConfig(jedisPoolConfig);
        JedisClientConfiguration.JedisClientConfigurationBuilder jedisClientConfigurationBuilder = JedisClientConfiguration.builder();
        jedisClientConfigurationBuilder.connectTimeout(redisProperties.getTimeout());//  connection timeout
        return jpcb.build();
    }

    @Bean
    @Scope(scopeName = "prototype")
    public RedisConnection redisConnection(RedisConnectionFactory factory) {
        return factory.getConnection();
    }

注意

Springboot提供的配置属性,与Jedis要求的不是很一致,需要自己配置,比如最大连接数,Jedis需要maxTotal,而Springboot中没有提供,所以需要在RedisConfiguration中自己根据实际情况配置.

RedisApplication

@SpringBootApplication
public class RedisApplication implements CommandLineRunner {
    @Autowired
    private RedisConnection redisConnection;

    public static void main(String[] args) {
        SpringApplication.run(RedisApplication.class, args);
    }

    @SuppressWarnings("all")
    public void run(String... strings) {
        redisConnection.set("datetime".getBytes(), LocalDateTime.now().toString().getBytes());
        String s = new String(redisConnection.get("datetime".getBytes()), StandardCharsets.UTF_8);
        System.out.println(s);
    }
}

总结

常用命令

meet

redis-cli -h ${host} -p ${port} CLUSTER meet ${another_host} ${another_port}

删除节点

redis-cli -h ${host} -p ${port} CLUSTER FORGET ${NODE_ID}

新版(5.0.7)redis-cli提供了很多关于集群的操作,可以通过CLUSTER HELP查看.

---

1. 现在如果再使用redis-trib.rb来搭建会直接报错并建议使用redis-cli命令. ↩︎