docker部署redis cluster模式

之前我们演示过如何在docker中部署redis哨兵模式，本文讲解如何在docker部署redis cluster模式

Redis Cluster着重于提高并发量。

群集至少需要3主3从，且每个实例使用不同的配置文件。

在redis-cluster架构中，redis-master节点一般用于接收读写，而redis-slave节点则一般只用于备份， 其与对应的master拥有相同的slot集合，若某个redis-master意外失效，则再将其对应的slave进行升级为临时redis-master。

在redis的官方文档中，对redis-cluster架构上，有这样的说明：在cluster架构下，默认的，一般redis-master用于接收读写，而redis-slave则用于备份，当有请求是在向slave发起时，会直接重定向到对应key所在的master来处理。 但如果不介意读取的是redis-cluster中有可能过期的数据并且对写请求不感兴趣时，则亦可通过readonly命令，将slave设置成可读，然后通过slave获取相关的key，达到读写分离。具体可以参阅redis官方文档等相关内容

本文中我们也设定6个节点，3主3从模式。设定的端口为7001~7006。首先在home目录下创建一个文件夹为redis，然后在创建一个目录为redis-cluster，在redis-cluster中创建6个文件夹，6个文件里都含有conf和data两个文件夹，然后在conf文件夹新建redis.conf,内容如下

# redis端口    #7001文件夹对应7001  7002对应7002。。。
port 7001    
# 关闭保护模式
protected-mode no   #关闭protected-mode模式，此时外部网络可以直接访问
# 开启集群
cluster-enabled yes
# 集群节点配置
cluster-config-file nodes.conf
# 超时
cluster-node-timeout 5000
# 集群节点IP host模式为宿主机IP
cluster-announce-ip 宿主机IP
# 集群节点端口 7001 - 7006
cluster-announce-port 7001     #7001文件夹对应7001  7002对应7002。。。
cluster-announce-bus-port 17001   #7001文件夹对应17001  7002对应17002。。。
# 开启 appendonly 备份模式
appendonly yes
# 每秒钟备份
appendfsync everysec
# 对aof文件进行压缩时，是否执行同步操作
no-appendfsync-on-rewrite no
# 当目前aof文件大小超过上一次重写时的aof文件大小的100%时会再次进行重写
auto-aof-rewrite-percentage 100
# 重写前AOF文件的大小最小值 默认 64mb
auto-aof-rewrite-min-size 64mb
requirepass  密码  #redis设定的密码
masterauth 884127 ##设定的密码

 6个redis.conf都设置完成后，回到redis目录中，新建docker-compose.yml，这里我们用docker-compose进行测试环境的docker编排。

version: '3.7'

services:
  redis7001:
    image: 'redis'
    container_name: redis7001
    restart: always
    command:
      ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-cluster/7001/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - ./redis-cluster/7001/data:/data
    ports:
      - "7001:7001"
      - "17001:17001"
    environment:
      # 设置时区为上海，否则时间会有问题
      - TZ=Asia/Shanghai


  redis7002:
    image: 'redis'
    container_name: redis7002
    restart: always
    command:
      ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-cluster/7002/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - ./redis-cluster/7002/data:/data
    ports:
      - "7002:7002"
      - "17002:17002"
    environment:
      # 设置时区为上海，否则时间会有问题
      - TZ=Asia/Shanghai


  redis7003:
    image: 'redis'
    container_name: redis7003
    restart: always
    command:
      ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-cluster/7003/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - ./redis-cluster/7003/data:/data
    ports:
      - "7003:7003"
      - "17003:17003"
    environment:
      # 设置时区为上海，否则时间会有问题
      - TZ=Asia/Shanghai


  redis7004:
    image: 'redis'
    container_name: redis7004
    restart: always
    command:
      ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-cluster/7004/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - ./redis-cluster/7004/data:/data
    ports:
      - "7004:7004"
      - "17004:17004"
    environment:
      # 设置时区为上海，否则时间会有问题
      - TZ=Asia/Shanghai


  redis7005:
    image: 'redis'
    container_name: redis7005
    restart: always
    command:
      ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-cluster/7005/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - ./redis-cluster/7005/data:/data
    ports:
      - "7005:7005"
      - "17005:17005"
    environment:
      # 设置时区为上海，否则时间会有问题
      - TZ=Asia/Shanghai


  redis7006:
    image: 'redis'
    container_name: redis7006
    restart: always
    command:
      ["redis-server", "/usr/local/etc/redis/redis.conf"]
    volumes:
      - ./redis-cluster/7006/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - ./redis-cluster/7006/data:/data
    ports:
      - "7006:7006"
      - "17006:17006"
    environment:
      # 设置时区为上海，否则时间会有问题
      - TZ=Asia/Shanghai

然后在redis目录中执行

[root@iZbp1bq6vb70qq3lbmjlh0Z redis]# docker-compose up -d
Creating network "redis_default" with the default driver
Creating redis7001 ... done
Creating redis7003 ... done
Creating redis7005 ... done
Creating redis7006 ... done
Creating redis7002 ... done
Creating redis7004 ... done
[root@iZbp1bq6vb70qq3lbmjlh0Z redis]# docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED          STATUS          PORTS                                                                                                NAMES
55e24cd0f4d1   redis                        "docker-entrypoint.s…"   5 seconds ago    Up 4 seconds    0.0.0.0:7004->7004/tcp, :::7004->7004/tcp, 6379/tcp, 0.0.0.0:17004->17004/tcp, :::17004->17004/tcp   redis7004
774eb9a83795   redis                        "docker-entrypoint.s…"   5 seconds ago    Up 4 seconds    0.0.0.0:7002->7002/tcp, :::7002->7002/tcp, 6379/tcp, 0.0.0.0:17002->17002/tcp, :::17002->17002/tcp   redis7002
41aa5da92d3f   redis                        "docker-entrypoint.s…"   5 seconds ago    Up 4 seconds    0.0.0.0:7005->7005/tcp, :::7005->7005/tcp, 6379/tcp, 0.0.0.0:17005->17005/tcp, :::17005->17005/tcp   redis7005
310810eeb305   redis                        "docker-entrypoint.s…"   5 seconds ago    Up 4 seconds    0.0.0.0:7003->7003/tcp, :::7003->7003/tcp, 6379/tcp, 0.0.0.0:17003->17003/tcp, :::17003->17003/tcp   redis7003
1c084aaa8490   redis                        "docker-entrypoint.s…"   5 seconds ago    Up 4 seconds    0.0.0.0:7006->7006/tcp, :::7006->7006/tcp, 6379/tcp, 0.0.0.0:17006->17006/tcp, :::17006->17006/tcp   redis7006
7140f0109668   redis                        "docker-entrypoint.s…"   5 seconds ago    Up 4 seconds    0.0.0.0:7001->7001/tcp, :::7001->7001/tcp, 6379/tcp, 0.0.0.0:17001->17001/tcp, :::17001->17001/tcp   redis7001

接下来我们利用redis-cli工具进行集群设置。

首先我们进入redis7001容器中，执行以下命令。

#密码就是redis.conf中设定的密码
docker exec -it redis7001 redis-cli  -p 7001 -a 密码  --cluster create 服务器ip:7001 服务器ip:7002 服务器ip:7003 服务器ip:7004 服务器ip:7005 服务器ip:7006 --cluster-replicas 1

然后就会出现以下信息。

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 服务器ip:7005 to 服务器ip:7001
Adding replica 服务器ip:7006 to 服务器ip:7002
Adding replica 服务器ip:7004 to 服务器ip:7003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001
   slots:[0-5460] (5461 slots) master
M: eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002
   slots:[5461-10922] (5462 slots) master
M: d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003
   slots:[10923-16383] (5461 slots) master
S: 5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004
   replicates 47d1bb69797150e405ae6346c6be50dce994164c
S: b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005
   replicates eab3edfab399b1a4d8a7d0232633a256adcac670
S: 4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006
   replicates d205a3e1175573dfcc11f5da5781c4910af7c17b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 服务器ip:7001)
M: 47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004
   slots: (0 slots) slave
   replicates 47d1bb69797150e405ae6346c6be50dce994164c
M: d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005
   slots: (0 slots) slave
   replicates eab3edfab399b1a4d8a7d0232633a256adcac670
S: 4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006
   slots: (0 slots) slave
   replicates d205a3e1175573dfcc11f5da5781c4910af7c17b
M: eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

集群测试

接下来进行一些集群的基本测试

1. 查看集群通信是否正常

redis7001主节点对它的副本节点redis7005进行ping操作。

docker exec -it redis7001 redis-cli -h 服务器ip -p 7005 -a 密码 ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG

2. 测试简单存储

redis7001主节点客户端操作redis7003主节点

docker docker exec -it redis7001 redis-cli -h 服务器ip -p 7003 -a 密码##  连接上redis7003
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
服务器ip:7003> set name admin
(error) MOVED 5798 服务器ip:7002    
服务器ip:7003> 


##由于Redis Cluster会根据key进行hash运算，然后将key分散到不同slots，name的hash运算结果在redis7002节点上的slots中。所以我们操作redis7003写操作，然后根据key运算，自动路由到7002。然而error提示无法路由？没关系，差一个 -c 参数而已。

#再次运行查看结果如下:

docker docker exec -it redis7001 redis-cli -h 服务器ip -p 7003 -a 密码 -c    

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
服务器ip:7003> set name admin
-> Redirected to slot [5798] located at 服务器ip:7002
OK
服务器ip:7002> get name
"admin"

3. 查看集群状态

服务器ip:7002> cluster nodes
541e37fe6c4c1719305ae58236d48a5dcdc2cde5 服务器ip:7002@17002 myself,master - 0 1637763650000 2 connected 5461-10922
3d215963e0d33153ffcaaac42cfc4f1e32040c5f 服务器ip:7004@17004 slave 0077c19f7a8351a578e8902e64b86590be2bf3e7 0 1637763651000 1 connected
825f9cbf6e718d6d290ae106a364a5eb748bd316 服务器ip:7005@17005 slave 541e37fe6c4c1719305ae58236d48a5dcdc2cde5 0 1637763651624 2 connected
2151df09dff7aea50eb522ff92180c9a138e050e 服务器ip:7003@17003 master - 0 1637763650000 3 connected 10923-16383
9ff4c4e95da28b301d408a99d1bedd5d9a06a93a 服务器ip:7006@17006 slave 2151df09dff7aea50eb522ff92180c9a138e050e 0 1637763651726 3 connected
0077c19f7a8351a578e8902e64b86590be2bf3e7 服务器ip:7001@17001 master - 0 1637763652230 1 connected 0-5460

4. 测试读写分离

服务器ip:7002> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=服务器ip,port=7005,state=online,offset=589,lag=0
master_failover_state:no-failover
master_replid:007793c00ebc6c4378724d0c2ef8db558effd1e2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:589
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:589

得到redis7002节点对应的从节点为redis7005，进入7005 获取刚刚在redis7002设定的key为name的值。

[root@iZbp1bq6vb70qq3lbmjlh0Z redis]#  docker exec -it redis7001 redis-cli -h 服务器ip -p 7005 -a 密码
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
服务器ip :7005> get name
(error) MOVED 5798 服务器ip :7002


##发现读不到，原来在redis cluster中，如果你要在slave读取数据，那么需要带先执行 readonly 指令，然后 get key



服务器ip :7005> readonly
OK
服务器ip :7005> get name
"admin"

容灾演练

现在我们杀掉主节点redis7002，看从节点redis7005是否会接替它的位置。

redis7002未杀掉之前，查看节点状态。

服务器ip:7005> cluster nodes
4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006@17006 slave d205a3e1175573dfcc11f5da5781c4910af7c17b 0 1637807788000 3 connected
5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004@17004 slave 47d1bb69797150e405ae6346c6be50dce994164c 0 1637807788569 1 connected
eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002@17002 master - 0 1637807789875 2 connected 5461-10922
d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003@17003 master - 0 1637807788871 3 connected 10923-16383
b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005@17005 myself,slave eab3edfab399b1a4d8a7d0232633a256adcac670 0 1637807789000 2 connected
47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001@17001 master - 0 1637807788000 1 connected 0-5460

杀掉之后

服务器ip:7005> cluster nodes
4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006@17006 slave d205a3e1175573dfcc11f5da5781c4910af7c17b 0 1637808034000 3 connected
5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004@17004 slave 47d1bb69797150e405ae6346c6be50dce994164c 0 1637808034609 1 connected
eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002@17002 master,fail - 1637808010507 1637808008000 2 disconnected
d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003@17003 master - 0 1637808034911 3 connected 10923-16383
b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005@17005 myself,master - 0 1637808033000 7 connected 5461-10922

##看到7005节点变成master

47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001@17001 master - 0 1637808034000 1 connected 0-5460

redis7002节点恢复后 就会变成7005的从节点

服务器ip:7005> cluster nodes
4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006@17006 slave d205a3e1175573dfcc11f5da5781c4910af7c17b 0 1637808081174 3 connected
5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004@17004 slave 47d1bb69797150e405ae6346c6be50dce994164c 0 1637808082179 1 connected
eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002@17002 slave b54e2fc1ca06a6a1b2a24ade5059420d988e0081 0 1637808081000 7 connected
d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003@17003 master - 0 1637808082000 3 connected 10923-16383
b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005@17005 myself,master - 0 1637808080000 7 connected 5461-10922
47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001@17001 master - 0 1637808082000 1 connected 0-5460