docker compose搭建Redis Cluster集群环境

环境

为了让环境更加真实，本文使用多机环境：

• 192.168.124.3
• 192.168.124.4

搭建

整体搭建步骤主要分为以下几步：

• 下载 Redis 镜像（其实这步可以省略，因为创建容器时，如果本地镜像不存在，就会去远程拉取）；
• 编写 Redis 配置文件；
• 编写 Docker Compose 模板文件；
• 创建并启动所有服务容器；
• 创建 Redis Cluster 集群。

编写 Redis 配置文件

创建目录及文件
分别在 192.168.124.3 和 192.168.124.4两台机器上执行以下操作。

# 创建目录
mkdir -p /usr/local/docker-redis/redis-cluster
# 切换至指定目录
cd /usr/local/docker-redis/redis-cluster/
# 编写 redis-cluster.tmp
vi redis-cluster.tmpl

编写配置文件

192.168.124.3 机器的 redis-cluster.tmpl 文件内容如下：

port ${PORT}
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.124.3
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}

192.168.124.4 机器的 redis-cluster.tmpl 文件内容如下：

port ${PORT}
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.124.4
cluster-announce-port ${PORT}
cluster-announce-bus-port 1${PORT}

• port：节点端口；
• requirepass：添加访问认证；
• masterauth：如果主节点开启了访问认证，从节点访问主节点需要认证；
• protected-mode：保护模式，默认值yes。开启保护模式以后，需配置 bind ip 或者设置访问密码；关闭保护模式，外部网络可以直接访问；
• daemonize：是否以守护线程的方式启动（后台启动），默认no；
• appendonly：是否开启 AOF 持久化模式，默认no；
• cluster-enabled：是否开启集群模式，默认no；
• cluster-config-file：集群节点信息文件；
• cluster-config-timeout：集群节点连接超时时间；
• cluster-announce-ip：集群节点ip，填写宿主机IP；
• cluster-announce-port：集群节点映射端口；
• cluster-announce-bus-port：集群节点总线端口；
  在 192.168.124.3 机器的 redis-cluster 目录下执行以下命令：
  注意这里直接复制会出错，需要一条一条复制

for port in `seq 6371 6373`; do \
	mkdir -p ${port}/conf \
	&& PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
	&& mkdir -p ${port}/data;\
done

在 192.168.124.4 机器的 redis-cluster 目录下执行以下命令：

for port in `seq 6374 6376`; do \
	mkdir -p ${port}/conf \
	&& PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
	&& mkdir -p ${port}/data;\
done

在 192.168.124.3 机器执行查看命令结果如下，如果没有 tree 命令先安装 yum install -y tree 。

[root@localhost redis-cluster]# tree 
.
├── 6371
│   ├── conf
│   │   └── redis.conf
│   └── data
├── 6372
│   ├── conf
│   │   └── redis.conf
│   └── data
├── 6373
│   ├── conf
│   │   └── redis.conf
│   └── data
└── redis-cluster.tmpl

9 directories, 4 files

编写 Docker Compose 模板文件

在 192.168.124.3 机器的 /usr/local/docker-redis/redis-cluster 目录下创建 docker-compose.yml 文件并编辑。

# 描述 Compose 文件的版本信息
version: "3.8"
# 定义服务，可以多个
services:
    redis-6371: # 服务名称
      image: redis # 创建容器时所需的镜像
      container_name: redis-6371 # 容器名称
      restart: always # 容器总是重新启动
      network_mode: "host" # host 网络模式
      volumes: # 数据卷，目录挂载
        - /usr/local/docker-redis/redis-cluster/6371/conf/redis.conf:/usr/local/etc/redis/redis.conf
        - /usr/local/docker-redis/redis-cluster/6371/data:/data
      command: redis-server /usr/local/etc/redis/redis.conf # 覆盖容器启动后默认执行的命令

    redis-6372:
      image: redis
      container_name: redis-6372
      network_mode: "host"
      volumes:
        - /usr/local/docker-redis/redis-cluster/6372/conf/redis.conf:/usr/local/etc/redis/redis.conf
        - /usr/local/docker-redis/redis-cluster/6372/data:/data
      command: redis-server /usr/local/etc/redis/redis.conf

    redis-6373:
      image: redis
      container_name: redis-6373
      network_mode: "host"
      volumes:
        - /usr/local/docker-redis/redis-cluster/6373/conf/redis.conf:/usr/local/etc/redis/redis.conf
        - /usr/local/docker-redis/redis-cluster/6373/data:/data
      command: redis-server /usr/local/etc/redis/redis.conf

在 192.168.124.4 机器的 /usr/local/docker-redis/redis-cluster 目录下创建 docker-compose.yml 文件并编辑。

# 描述 Compose 文件的版本信息
version: "3.8"
# 定义服务，可以多个
services:
  redis-6374: # 服务名称
    image: redis # 创建容器时所需的镜像
    container_name: redis-6374 # 容器名称
    restart: always # 容器总是重新启动
    network_mode: "host" # host 网络模式
    volumes: # 数据卷，目录挂载
      - /usr/local/docker-redis/redis-cluster/6374/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - /usr/local/docker-redis/redis-cluster/6374/data:/data
    command: redis-server /usr/local/etc/redis/redis.conf # 覆盖容器启动后默认执行的命令

  redis-6375:
    image: redis
    container_name: redis-6375
    network_mode: "host"
    volumes:
      - /usr/local/docker-redis/redis-cluster/6375/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - /usr/local/docker-redis/redis-cluster/6375/data:/data
    command: redis-server /usr/local/etc/redis/redis.conf

  redis-6376:
    image: redis
    container_name: redis-6376
    network_mode: "host"
    volumes:
      - /usr/local/docker-redis/redis-cluster/6376/conf/redis.conf:/usr/local/etc/redis/redis.conf
      - /usr/local/docker-redis/redis-cluster/6376/data:/data
    command: redis-server /usr/local/etc/redis/redis.conf

创建并启动所有服务容器

在 192.168.124.3和 192.168.124.4机器的 /usr/local/docker-redis/redis-cluster 目录下执行以下
命令：

docker-compose up -d

[root@localhost redis-cluster]# docker-compose up -d
[+] Running 3/3
 ⠿ Container redis-6373  Started                                                                    0.7s
 ⠿ Container redis-6371  Started                                                                    0.7s
 ⠿ Container redis-6372  Started  

[root@localhost redis-cluster]# docker-compose up -d
[+] Running 3/3
 ⠿ Container redis-6375  Started                                                                    0.5s
 ⠿ Container redis-6374  Started                                                                    0.5s
 ⠿ Container redis-6376  Started   

创建 Redis Cluster 集群

随便进入一个容器节点，并进入 /usr/local/bin/ 目录：

# 进入容器
docker exec -it redis-6371 bash
# 切换至指定目录
cd /usr/local/bin/

接下来我们就可以通过以下命令实现 Redis Cluster 集群的创建。

redis-cli -a 1234 --cluster create 192.168.124.3:6371 192.168.124.3:6372 192.168.124.3:6373 192.168.124.4:6374 192.168.124.4:6375 192.168.124.4:6376 --cluster-replicas 1

出现选择提示信息，输入yes

Can I set the above configuration? (type 'yes' to accept): 

集群创建成功如下：

root@localhost:/usr/local/bin# redis-cli -a 1234 --cluster create 192.168.124.3:6371 192.168.124.3:6372 192.168.124.3:6373 192.168.124.4:6374 192.168.124.4.11:6375 192.168.124.4:6376 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Could not connect to Redis at 192.168.124.4.11:6375: Name or service not known
root@localhost:/usr/local/bin# redis-cli -a 1234 --cluster create 192.168.124.3:6371 192.168.124.3:6372 192.168.124.3:6373 192.168.124.4:6374 192.168.124.4:6375 192.168.124.4:6376 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.124.4:6376 to 192.168.124.3:6371
Adding replica 192.168.124.3:6373 to 192.168.124.4:6374
Adding replica 192.168.124.4:6375 to 192.168.124.3:6372
M: 1e0845fa94b0cc2c4ed1fc70acba3576bd46580c 192.168.124.3:6371
   slots:[0-5460] (5461 slots) master
M: 474937aa8eb7fee3bf947092207e2a31fcd85f70 192.168.124.3:6372
   slots:[10923-16383] (5461 slots) master
S: 0785a89b37da34bf4def95afe6ba847afa6f2420 192.168.124.3:6373
   replicates 53483ccc379e0108acfebd8c9d89e7c3b273c4af
M: 53483ccc379e0108acfebd8c9d89e7c3b273c4af 192.168.124.4:6374
   slots:[5461-10922] (5462 slots) master
S: 3979d390a1b3b1f67a065e8eb511449cbd184b07 192.168.124.4:6375
   replicates 474937aa8eb7fee3bf947092207e2a31fcd85f70
S: b3fd8bc0806298b4a9533c2df28fae03e9d7888c 192.168.124.4:6376
   replicates 1e0845fa94b0cc2c4ed1fc70acba3576bd46580c
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.124.3:6371)
M: 1e0845fa94b0cc2c4ed1fc70acba3576bd46580c 192.168.124.3:6371
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 53483ccc379e0108acfebd8c9d89e7c3b273c4af 192.168.124.4:6374
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 0785a89b37da34bf4def95afe6ba847afa6f2420 192.168.124.3:6373
   slots: (0 slots) slave
   replicates 53483ccc379e0108acfebd8c9d89e7c3b273c4af
S: 3979d390a1b3b1f67a065e8eb511449cbd184b07 192.168.124.4:6375
   slots: (0 slots) slave
   replicates 474937aa8eb7fee3bf947092207e2a31fcd85f70
M: 474937aa8eb7fee3bf947092207e2a31fcd85f70 192.168.124.3:6372
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: b3fd8bc0806298b4a9533c2df28fae03e9d7888c 192.168.124.4:6376
   slots: (0 slots) slave
   replicates 1e0845fa94b0cc2c4ed1fc70acba3576bd46580c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

至此一个高可用的 Redis Cluster 集群搭建完成，如下图所示，该集群中包含 6 个 Redis 节点，3主 3 从。三个主节点会分配槽，处理客户端的命令请求，而从节点可用在主节点故障后，顶替主节点。

查看集群状态

先进入容器，然后通过一些集群常用的命令查看一下集群的状态。

# 进入容器
docker exec -it redis-6371 bash
# 切换至指定目录
cd /usr/local/bin/

检查集群状态

redis-cli -a 1234 --cluster check 192.168.124.3:6371

root@localhost:/usr/local/bin# redis-cli -a 1234 --cluster check 192.168.124.3:6371
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.124.3:6371 (1e0845fa...) -> 0 keys | 5461 slots | 1 slaves.
192.168.124.4:6374 (53483ccc...) -> 0 keys | 5462 slots | 1 slaves.
192.168.124.3:6372 (474937aa...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.124.3:6371)
M: 1e0845fa94b0cc2c4ed1fc70acba3576bd46580c 192.168.124.3:6371
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 53483ccc379e0108acfebd8c9d89e7c3b273c4af 192.168.124.4:6374
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 0785a89b37da34bf4def95afe6ba847afa6f2420 192.168.124.3:6373
   slots: (0 slots) slave
   replicates 53483ccc379e0108acfebd8c9d89e7c3b273c4af
S: 3979d390a1b3b1f67a065e8eb511449cbd184b07 192.168.124.4:6375
   slots: (0 slots) slave
   replicates 474937aa8eb7fee3bf947092207e2a31fcd85f70
M: 474937aa8eb7fee3bf947092207e2a31fcd85f70 192.168.124.3:6372
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: b3fd8bc0806298b4a9533c2df28fae03e9d7888c 192.168.124.4:6376
   slots: (0 slots) slave
   replicates 1e0845fa94b0cc2c4ed1fc70acba3576bd46580c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看集群信息和节点信息

# 连接至集群某个节点
redis-cli -c -a 1234 -h 192.168.124.4 -p 6374
# 查看集群信息
cluster info
# 查看集群结点信息
cluster nodes

192.168.124.4:6374> exit
root@localhost:/usr/local/bin# redis-cli -c -a 1234 -h 192.168.124.4 -p 6374
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.124.4:6374> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:4
cluster_stats_messages_ping_sent:537
cluster_stats_messages_pong_sent:534
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:1072
cluster_stats_messages_ping_received:534
cluster_stats_messages_pong_received:538
cluster_stats_messages_received:1072
192.168.124.4:6374> cluster nodes
474937aa8eb7fee3bf947092207e2a31fcd85f70 192.168.124.3:6372@16372 master - 0 1635065754467 2 connected 10923-16383
3979d390a1b3b1f67a065e8eb511449cbd184b07 192.168.124.4:6375@16375 slave 474937aa8eb7fee3bf947092207e2a31fcd85f70 0 1635065751000 2 connected
1e0845fa94b0cc2c4ed1fc70acba3576bd46580c 192.168.124.3:6371@16371 master - 0 1635065752000 1 connected 0-5460
0785a89b37da34bf4def95afe6ba847afa6f2420 192.168.124.3:6373@16373 slave 53483ccc379e0108acfebd8c9d89e7c3b273c4af 0 1635065752451 4 connected
53483ccc379e0108acfebd8c9d89e7c3b273c4af 192.168.124.4:6374@16374 myself,master - 0 1635065753000 4 connected 5461-10922
b3fd8bc0806298b4a9533c2df28fae03e9d7888c 192.168.124.4:6376@16376 slave 1e0845fa94b0cc2c4ed1fc70acba3576bd46580c 0 1635065753458 1 connected

SET/GET
在 6371 节点中执行写入和读取，命令如下：

# 进入容器并连接至集群某个节点
docker exec -it redis-6371 /usr/local/bin/redis-cli -c -a 1234 -h 192.168.124.3 -p 6371
# 写入数据
set name oxyay
set a 1
set b 2
# 读取数据
get name
get a
get b

[root@localhost redis-cluster]# docker exec -it redis-6371 /usr/local/bin/redis-cli -c -a 1234 -h 192.168.124.3 -p 6371
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.124.3:6371> set name oxyay
-> Redirected to slot [5798] located at 192.168.124.4:6374
OK
192.168.124.4:6374> set a 1
-> Redirected to slot [15495] located at 192.168.124.3:6372
OK
192.168.124.3:6372> set b 2
-> Redirected to slot [3300] located at 192.168.124.3:6371
OK
192.168.124.3:6371> get name
-> Redirected to slot [5798] located at 192.168.124.4:6374
"oxyay"
192.168.124.4:6374> get a
-> Redirected to slot [15495] located at 192.168.124.3:6372
"1"
192.168.124.3:6372> get b
-> Redirected to slot [3300] located at 192.168.124.3:6371
"2"