Docker 部署 Redis Cluster 集群

Docker Image Version

redis:7.0.11-alpine

方案意义

为了解决 Redis 高性能高可用的问题，Redis 推出了自带的 Cluster 模式。他通过将一整个集群分成 16384 哈希槽位，再将槽位平均分配到每个主节点上面。同时主节点采用 Redis Replication 主从模式（自带 Sentinel 哨兵机制）防止主节点故障。同时 Redis 作者强烈不建议集群数目大于 1000 台节点，因为通信需要占用很大的带宽。

三主三从 Docker Compose 基础配置

version: '3.8'

services:
  redis-cluster1:
    container_name: redis-cluster1
    image: redis:7.0.11-alpine
    hostname: redis-cluster1
    restart: unless-stopped
    ports:
      - "6379:6379"
    networks:
      redis-cluster:
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: '3'
    volumes:
      - redis-cluster1:/data
    command:
      - redis-server
      - --bind
      - "0.0.0.0"
      - --port
      - "6379"
      - --requirepass
      - "ChangeMe"
      - --appendonly
      - "yes"
      - --masterauth
      - "ChangeMe"
      - --cluster-enabled
      - "yes"
      - --cluster-config-file
      - "redis-cluster1.conf"
        
  redis-cluster2:
    container_name: redis-cluster2
    image: redis:7.0.11-alpine
    hostname: redis-cluster2
    restart: unless-stopped
    ports:
      - "6380:6379"
    networks:
      redis-cluster:
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: '3'
    volumes:
      - redis-cluster2:/data
    command:
      - redis-server
      - --bind
      - "0.0.0.0"
      - --port
      - "6379"
      - --requirepass
      - "ChangeMe"
      - --appendonly
      - "yes"
      - --masterauth
      - "ChangeMe"
      - --cluster-enabled
      - "yes"
      - --cluster-config-file
      - "redis-cluster2.conf"

  redis-cluster3:
    container_name: redis-cluster3
    image: redis:7.0.11-alpine
    hostname: redis-cluster3
    restart: unless-stopped
    ports:
      - "6381:6379"
    networks:
      redis-cluster:
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: '3'
    volumes:
      - redis-cluster3:/data
    command:
      - redis-server
      - --bind
      - "0.0.0.0"
      - --port
      - "6379"
      - --requirepass
      - "ChangeMe"
      - --appendonly
      - "yes"
      - --masterauth
      - "ChangeMe"
      - --cluster-enabled
      - "yes"
      - --cluster-config-file
      - "redis-cluster3.conf"
        
  redis-cluster4:
    container_name: redis-cluster4
    image: redis:7.0.11-alpine
    hostname: redis-cluster4
    restart: unless-stopped
    ports:
      - "6382:6379"
    networks:
      redis-cluster:
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: '3'
    volumes:
      - redis-cluster4:/data
    command:
      - redis-server
      - --bind
      - "0.0.0.0"
      - --port
      - "6379"
      - --requirepass
      - "ChangeMe"
      - --appendonly
      - "yes"
      - --masterauth
      - "ChangeMe"
      - --cluster-enabled
      - "yes"
      - --cluster-config-file
      - "redis-cluster4.conf"

  redis-cluster5:
    container_name: redis-cluster5
    image: redis:7.0.11-alpine
    hostname: redis-cluster5
    restart: unless-stopped
    ports:
      - "6383:6379"
    networks:
      redis-cluster:
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: '3'
    volumes:
      - redis-cluster5:/data
    command:
      - redis-server
      - --bind
      - "0.0.0.0"
      - --port
      - "6379"
      - --requirepass
      - "ChangeMe"
      - --appendonly
      - "yes"
      - --masterauth
      - "ChangeMe"
      - --cluster-enabled
      - "yes"
      - --cluster-config-file
      - "redis-cluster5.conf"

  redis-cluster6:
    container_name: redis-cluster6
    image: redis:7.0.11-alpine
    hostname: redis-cluster6
    restart: unless-stopped
    ports:
      - "6384:6379"
    networks:
      redis-cluster:
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: '3'
    volumes:
      - redis-cluster6:/data
    command:
      - redis-server
      - --bind
      - "0.0.0.0"
      - --port
      - "6379"
      - --requirepass
      - "ChangeMe"
      - --appendonly
      - "yes"
      - --masterauth
      - "ChangeMe"
      - --cluster-enabled
      - "yes"
      - --cluster-config-file
      - "redis-cluster6.conf"


volumes:
  redis-cluster1:
  redis-cluster2:
  redis-cluster3:
  redis-cluster4:
  redis-cluster5:
  redis-cluster6:

networks:
  redis-cluster:
    driver: bridge
    ipam:
      config:
        - subnet: 172.21.0.0/16

创建 Cluster 集群

docker exec -it redis-cluster1 redis-cli -a ChangeMe --cluster create --cluster-replicas 1 redis-cluster1:6379 redis-cluster2:6379 redis-cluster3:6379 redis-cluster4:6379 redis-cluster5:6379 redis-cluster6:6379

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica redis-cluster5:6379 to redis-cluster1:6379
Adding replica redis-cluster6:6379 to redis-cluster2:6379
Adding replica redis-cluster4:6379 to redis-cluster3:6379
M: 5e262c16f666affcd260d5688242d993194b86d7 redis-cluster1:6379
   slots:[0-5460] (5461 slots) master
M: 0f9955ecb806141e8149bf63a7ca9ffb6bc73918 redis-cluster2:6379
   slots:[5461-10922] (5462 slots) master
M: 63b7152d0af00bcd55a4b7ac3811104ae810fef5 redis-cluster3:6379
   slots:[10923-16383] (5461 slots) master
S: 8d50b5b6e61e2eebdf80ca7b3a3d75858fc2b3a0 redis-cluster4:6379
   replicates 63b7152d0af00bcd55a4b7ac3811104ae810fef5
S: 0b7aa8453ab1607fdc4c4a893945f239b4da92fa redis-cluster5:6379
   replicates 5e262c16f666affcd260d5688242d993194b86d7
S: 210dceb99e1e3a0508ee9a0262afcdaaddfe46a9 redis-cluster6:6379
   replicates 0f9955ecb806141e8149bf63a7ca9ffb6bc73918
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node redis-cluster1:6379)
M: 5e262c16f666affcd260d5688242d993194b86d7 redis-cluster1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 0f9955ecb806141e8149bf63a7ca9ffb6bc73918 172.21.0.5:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 8d50b5b6e61e2eebdf80ca7b3a3d75858fc2b3a0 172.21.0.7:6379
   slots: (0 slots) slave
   replicates 63b7152d0af00bcd55a4b7ac3811104ae810fef5
S: 210dceb99e1e3a0508ee9a0262afcdaaddfe46a9 172.21.0.2:6379
   slots: (0 slots) slave
   replicates 0f9955ecb806141e8149bf63a7ca9ffb6bc73918
M: 63b7152d0af00bcd55a4b7ac3811104ae810fef5 172.21.0.4:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 0b7aa8453ab1607fdc4c4a893945f239b4da92fa 172.21.0.6:6379
   slots: (0 slots) slave
   replicates 5e262c16f666affcd260d5688242d993194b86d7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

集群查询

docker exec -it redis-cluster1 redis-cli -a ChangeMe -p 6379 cluster nodes

0f9955ecb806141e8149bf63a7ca9ffb6bc73918 172.21.0.5:6379@16379 master - 0 1687404017000 2 connected 5461-10922
8d50b5b6e61e2eebdf80ca7b3a3d75858fc2b3a0 172.21.0.7:6379@16379 slave 63b7152d0af00bcd55a4b7ac3811104ae810fef5 0 1687404018111 3 connected
210dceb99e1e3a0508ee9a0262afcdaaddfe46a9 172.21.0.2:6379@16379 slave 0f9955ecb806141e8149bf63a7ca9ffb6bc73918 0 1687404018000 2 connected
63b7152d0af00bcd55a4b7ac3811104ae810fef5 172.21.0.4:6379@16379 master - 0 1687404019136 3 connected 10923-16383
0b7aa8453ab1607fdc4c4a893945f239b4da92fa 172.21.0.6:6379@16379 slave 5e262c16f666affcd260d5688242d993194b86d7 0 1687404020165 1 connected
5e262c16f666affcd260d5688242d993194b86d7 172.21.0.3:6379@16379 myself,master - 0 1687404017000 1 connected 0-5460

集群水平扩展

启动redis-cluster7和redis-cluster8两台容器

docker run -d --hostname redis-cluster7 --name redis-cluster7 --log-driver=json-file \
--log-opt max-size=100m --log-opt max-file=3 --restart=unless-stopped \
-v redis-cluster7:/data -p 6385:6379 --network=redis_redis-cluster redis:7.0.11-alpine \
redis-server --requirepass ChangeMe --appendonly yes --cluster-enabled yes \
--cluster-config-file redis-cluster7.conf && \

docker run -d --hostname redis-cluster8 --name redis-cluster8 --log-driver=json-file \
--log-opt max-size=100m --log-opt max-file=3 --restart=unless-stopped \
-v redis-cluster8:/data -p 6386:6379 --network=redis_redis-cluster redis:7.0.11-alpine \
redis-server --requirepass ChangeMe --appendonly yes --cluster-enabled yes \
--cluster-config-file redis-cluster8.conf

将redis-cluster7作为主节点，redis-cluster8作为redis-cluster7的副节点加入集群

docker exec -it redis-cluster1 redis-cli -a ChangeMe -p 6379 --cluster add-node redis-cluster7:6379 redis-cluster1:6379

>>> Adding node redis-cluster7:6379 to cluster redis-cluster1:6379
>>> Performing Cluster Check (using node redis-cluster1:6379)
M: 5e262c16f666affcd260d5688242d993194b86d7 redis-cluster1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 0f9955ecb806141e8149bf63a7ca9ffb6bc73918 172.21.0.5:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 8d50b5b6e61e2eebdf80ca7b3a3d75858fc2b3a0 172.21.0.7:6379
   slots: (0 slots) slave
   replicates 63b7152d0af00bcd55a4b7ac3811104ae810fef5
S: 210dceb99e1e3a0508ee9a0262afcdaaddfe46a9 172.21.0.2:6379
   slots: (0 slots) slave
   replicates 0f9955ecb806141e8149bf63a7ca9ffb6bc73918
M: 63b7152d0af00bcd55a4b7ac3811104ae810fef5 172.21.0.4:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 0b7aa8453ab1607fdc4c4a893945f239b4da92fa 172.21.0.6:6379
   slots: (0 slots) slave
   replicates 5e262c16f666affcd260d5688242d993194b86d7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Getting functions from cluster
>>> Send FUNCTION LIST to redis-cluster7:6379 to verify there is no functions in it
>>> Send FUNCTION RESTORE to redis-cluster7:6379
>>> Send CLUSTER MEET to node redis-cluster7:6379 to make it join the cluster.
[OK] New node added correctly.

查询redis-cluster7的master id为3194b5c5820004232b351a9a9170a1481c34e31f

docker exec -it redis-cluster7 redis-cli -a ChangeMe -p 6379 cluster nodes

5e262c16f666affcd260d5688242d993194b86d7 172.21.0.3:6379@16379 master - 0 1687406681000 1 connected 0-5460
3194b5c5820004232b351a9a9170a1481c34e31f 172.21.0.8:6379@16379 myself,master - 0 1687406682000 0 connected
8d50b5b6e61e2eebdf80ca7b3a3d75858fc2b3a0 172.21.0.7:6379@16379 slave 63b7152d0af00bcd55a4b7ac3811104ae810fef5 0 1687406684006 3 connected
0f9955ecb806141e8149bf63a7ca9ffb6bc73918 172.21.0.5:6379@16379 master - 0 1687406680000 2 connected 5461-10922
210dceb99e1e3a0508ee9a0262afcdaaddfe46a9 172.21.0.2:6379@16379 slave 0f9955ecb806141e8149bf63a7ca9ffb6bc73918 0 1687406681946 2 connected
0b7aa8453ab1607fdc4c4a893945f239b4da92fa 172.21.0.6:6379@16379 slave 5e262c16f666affcd260d5688242d993194b86d7 0 1687406682972 1 connected
63b7152d0af00bcd55a4b7ac3811104ae810fef5 172.21.0.4:6379@16379 master - 0 1687406680920 3 connected 10923-16383

加入副节点redis-cluster8

docker exec -it redis-cluster1 redis-cli -a ChangeMe --cluster add-node redis-cluster8:6379 redis-cluster1:6379 --cluster-slave --cluster-master-id 3194b5c5820004232b351a9a9170a1481c34e31f

>>> Adding node redis-cluster8:6379 to cluster redis-cluster1:6379
>>> Performing Cluster Check (using node redis-cluster1:6379)
M: 5e262c16f666affcd260d5688242d993194b86d7 redis-cluster1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 3194b5c5820004232b351a9a9170a1481c34e31f 172.21.0.8:6379
   slots: (0 slots) master
M: 0f9955ecb806141e8149bf63a7ca9ffb6bc73918 172.21.0.5:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 8d50b5b6e61e2eebdf80ca7b3a3d75858fc2b3a0 172.21.0.7:6379
   slots: (0 slots) slave
   replicates 63b7152d0af00bcd55a4b7ac3811104ae810fef5
S: 210dceb99e1e3a0508ee9a0262afcdaaddfe46a9 172.21.0.2:6379
   slots: (0 slots) slave
   replicates 0f9955ecb806141e8149bf63a7ca9ffb6bc73918
M: 63b7152d0af00bcd55a4b7ac3811104ae810fef5 172.21.0.4:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 0b7aa8453ab1607fdc4c4a893945f239b4da92fa 172.21.0.6:6379
   slots: (0 slots) slave
   replicates 5e262c16f666affcd260d5688242d993194b86d7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node redis-cluster8:6379 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 172.21.0.8:6379.
[OK] New node added correctly.

重新分配卡槽

docker exec -it redis-cluster1 redis-cli -a ChangeMe --cluster reshard redis-cluster1:6379

>>> Performing Cluster Check (using node redis-cluster1:6379)
M: 5e262c16f666affcd260d5688242d993194b86d7 redis-cluster1:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
M: 3194b5c5820004232b351a9a9170a1481c34e31f 172.21.0.8:6379
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
   1 additional replica(s)
M: 0f9955ecb806141e8149bf63a7ca9ffb6bc73918 172.21.0.5:6379
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 8d50b5b6e61e2eebdf80ca7b3a3d75858fc2b3a0 172.21.0.7:6379
   slots: (0 slots) slave
   replicates 63b7152d0af00bcd55a4b7ac3811104ae810fef5
S: 210dceb99e1e3a0508ee9a0262afcdaaddfe46a9 172.21.0.2:6379
   slots: (0 slots) slave
   replicates 0f9955ecb806141e8149bf63a7ca9ffb6bc73918
S: 5ca1b4875a404334a729266ad9e9ffa9a361135e 172.21.0.9:6379
   slots: (0 slots) slave
   replicates 3194b5c5820004232b351a9a9170a1481c34e31f
M: 63b7152d0af00bcd55a4b7ac3811104ae810fef5 172.21.0.4:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 0b7aa8453ab1607fdc4c4a893945f239b4da92fa 172.21.0.6:6379
   slots: (0 slots) slave
   replicates 5e262c16f666affcd260d5688242d993194b86d7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 3194b5c5820004232b351a9a9170a1481c34e31f
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: 
...

这里可以指定固定的源节点，然后回车输入done，同时也可以输入all指定所有的源节点进行迁移。