Docker搭建Redis Cluster集群环境

编写 Redis 配置文件

创建目录及文件夹

分别在 192.168.124.3 和 192.168,124.4 两台机器上执行以下操作。

# 创建目录
mkdir -p /usr/local/docker-redis/redis-cluster
# 切换至指定目录
cd /usr/local/docker-redis/redis-cluster
# 编写 redis-cluster.tmpl 文件
vi redis-cluster.tmpl

编写配置文件

192.168.124.3 机器的 redis-cluster.tmpl 文件内容如下：

port ${PORT}
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.124.3
cluster-announce-port ${PORT}
cluster-announce-bus-port  1${PORT}

192.168.124.4 机器的 redis-cluster.tmpl 文件内容如下：

port ${PORT}
requirepass 1234
masterauth 1234
protected-mode no
daemonize no
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 192.168.124.4
cluster-announce-port ${PORT}
cluster-announce-bus-port  1${PORT}

• port：节点端口；
• requirepass：添加访问认证；
• masterauth：如果主节点开启了访问认证，从节点访问主节点需要认证；
• protected-mode：保护模式，默认值yes。开启保护模式以后，需配置 bind ip 或者设置访问密码；关闭保护模式，外部网络可以直接访问；
• daemonize：是否以守护线程的方式启动（后台启动），默认no；
• appendonly：是否开启 AOF 持久化模式，默认no；
• cluster-enabled：是否开启集群模式，默认no；
• cluster-config-file：集群节点信息文件；
• cluster-config-timeout：集群节点连接超时时间；
• cluster-announce-ip：集群节点ip，填写宿主机IP；
• cluster-announce-port：集群节点映射端口；
• cluster-announce-bus-port：集群节点总线端口；

在 192.168.124.3 机器的 redis-cluster 目录下执行以下命令：

for port in `seq 6374 6376`; do \
mkdir -p ${port}/conf \
&& PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
&& mkdir -p ${port}/data;\
done

在 192.168.124.4 机器的 redis-cluster 目录下执行以下命令：

for port in `seq 6371 6373`; do \
mkdir -p ${port}/conf \
&& PORT=${port} envsubst < redis-cluster.tmpl > ${port}/conf/redis.conf \
&& mkdir -p ${port}/data;\
done

在 192.168.124.3 机器执行查看命令结果。

[root@localhost redis-cluster]# tree
.
├── 6374
│   ├── conf
│   │   └── redis.conf
│   └── data
├── 6375
│   ├── conf
│   │   └── redis.conf
│   └── data
├── 6376
│   ├── conf
│   │   └── redis.conf
│   └── data
└── redis-cluster.tmpl

9 directories, 4 files

在 192.168.124.4 机器执行查看命令结果。

[root@localhost redis-cluster]# tree
.
├── 6371
│   ├── conf
│   │   └── redis.conf
│   └── data
├── 6372
│   ├── conf
│   │   └── redis.conf
│   └── data
├── 6373
│   ├── conf
│   │   └── redis.conf
│   └── data
└── redis-cluster.tmpl

9 directories, 4 files

创建 Redis 容器

创建容器

将宿主机的 6371 ~ 6376 之间的端口与6个 Redis 容器映射，并将宿主机的目录与容器内的目录进行映射（目录挂载）。使用 host 网络模式。

在 192.168.124.3 机器执行以下命令：

for port in $(seq 6374 6376); do \
  docker run -id --restart always --name redis-${port} --net host \
  -v /usr/local/docker-redis/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
  -v /usr/local/docker-redis/redis-cluster/${port}/data:/data \
  redis redis-server /usr/local/etc/redis/redis.conf; \
done

在 192.168.124.4 机器执行以下命令：

for port in $(seq 6371 6373); do \
  docker run -id --restart always --name redis-${port} --net host \
  -v /usr/local/docker-redis/redis-cluster/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf \
  -v /usr/local/docker-redis/redis-cluster/${port}/data:/data \
  redis redis-server /usr/local/etc/redis/redis.conf; \
done

创建 Redis Cluster 集群

随便进入一个容器节点，并进入 /usr/local/bin/ 目录：

# 进入容器
docker exec -it redis-6371 bash
# 切换至指定目录
cd /usr/local/bin/

redis-cli -a 1234 --cluster create 192.168.124.4:6371 192.168.124.4:6372 192.168.124.4:6373 192.168.124.3:6374  192.168.124.3:6375 192.168.124.3:6376 --cluster-replicas 1

root@localhost:/usr/local/etc# redis-cli -a 1234 --cluster create 192.168.124.4:6371 192.168.124.4:6372 192.168.124.4:6373 192.168.124.3:6374  192.168.124.3:6375 192.168.124.3:6376 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.124.3:6376 to 192.168.124.4:6371
Adding replica 192.168.124.4:6373 to 192.168.124.3:6374
Adding replica 192.168.124.3:6375 to 192.168.124.4:6372
M: e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83 192.168.124.4:6371
   slots:[0-5460] (5461 slots) master
M: c17b158ebed623157f0ccde73993de775e325d0c 192.168.124.4:6372
   slots:[10923-16383] (5461 slots) master
S: 8b8dc2ca0d2b87ab31702abb0936ace9348be7e3 192.168.124.4:6373
   replicates bf3f52c03723f497d870e063a3569a321781c87d
M: bf3f52c03723f497d870e063a3569a321781c87d 192.168.124.3:6374
   slots:[5461-10922] (5462 slots) master
S: af7ea10aaf53d39a1c0b16d9b5e28d1b9cdddccf 192.168.124.3:6375
   replicates c17b158ebed623157f0ccde73993de775e325d0c
S: fe904b056b4502061015352983fff118d0009b97 192.168.124.3:6376
   replicates e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.124.4:6371)
M: e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83 192.168.124.4:6371
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 8b8dc2ca0d2b87ab31702abb0936ace9348be7e3 192.168.124.4:6373
   slots: (0 slots) slave
   replicates bf3f52c03723f497d870e063a3569a321781c87d
M: bf3f52c03723f497d870e063a3569a321781c87d 192.168.124.3:6374
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: c17b158ebed623157f0ccde73993de775e325d0c 192.168.124.4:6372
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: af7ea10aaf53d39a1c0b16d9b5e28d1b9cdddccf 192.168.124.3:6375
   slots: (0 slots) slave
   replicates c17b158ebed623157f0ccde73993de775e325d0c
S: fe904b056b4502061015352983fff118d0009b97 192.168.124.3:6376
   slots: (0 slots) slave
   replicates e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

可能会出现这种情况。

关闭另一台机器的防火墙。

[root@localhost redis-cluster]#  systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
[root@localhost redis-cluster]#  systemctl stop firewalld

查看集群的状态

先进入容器，通过集群的常用命令查看集群的状态。

# 进入容器
docker exec -it redis-6371 bash
# 切换至指定目录
cd /usr/local/bin/

检查集群状态

redis -a 1234 --cluster check 192.168.124.3:6374

root@localhost:/data# redis-cli -a 1234 --cluster check 192.168.124.3:6374
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.124.3:6374 (bf3f52c0...) -> 0 keys | 5462 slots | 1 slaves.
192.168.124.4:6372 (c17b158e...) -> 1 keys | 5461 slots | 1 slaves.
192.168.124.4:6371 (e0e62e31...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.124.3:6374)
M: bf3f52c03723f497d870e063a3569a321781c87d 192.168.124.3:6374
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: c17b158ebed623157f0ccde73993de775e325d0c 192.168.124.4:6372
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: fe904b056b4502061015352983fff118d0009b97 192.168.124.3:6376
   slots: (0 slots) slave
   replicates e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83
M: e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83 192.168.124.4:6371
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: af7ea10aaf53d39a1c0b16d9b5e28d1b9cdddccf 192.168.124.3:6375
   slots: (0 slots) slave
   replicates c17b158ebed623157f0ccde73993de775e325d0c
S: 8b8dc2ca0d2b87ab31702abb0936ace9348be7e3 192.168.124.4:6373
   slots: (0 slots) slave
   replicates bf3f52c03723f497d870e063a3569a321781c87d
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看集群信息和节点信息

# 连接至集群某个节点
redis-cli -c -a 1234 -h 192.168.124.3 -p 6374
# 查看集群信息
cluster info
# 查看集群结点信息
CLUSTER NODES

root@localhost:/data# redis-cli -c -a 1234 -h 192.168.124.3 -p 6374
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.124.3:6374> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:4
cluster_stats_messages_ping_sent:1558
cluster_stats_messages_pong_sent:1505
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:3064
cluster_stats_messages_ping_received:1505
cluster_stats_messages_pong_received:1559
cluster_stats_messages_received:3064
192.168.124.3:6374> CLUSTER NODES
bf3f52c03723f497d870e063a3569a321781c87d 192.168.124.3:6374@16374 myself,master - 0 1634901874000 4 connected 5461-10922
c17b158ebed623157f0ccde73993de775e325d0c 192.168.124.4:6372@16372 master - 0 1634901877050 2 connected 10923-16383
fe904b056b4502061015352983fff118d0009b97 192.168.124.3:6376@16376 slave e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83 0 1634901877000 1 connected
e0e62e31ae67c3c54ef4cf371ef09c5bcb36cd83 192.168.124.4:6371@16371 master - 0 1634901876041 1 connected 0-5460
af7ea10aaf53d39a1c0b16d9b5e28d1b9cdddccf 192.168.124.3:6375@16375 slave c17b158ebed623157f0ccde73993de775e325d0c 0 1634901876000 2 connected
8b8dc2ca0d2b87ab31702abb0936ace9348be7e3 192.168.124.4:6373@16373 slave bf3f52c03723f497d870e063a3569a321781c87d 0 1634901878060 4 connected

SET/GET

在6374节点中执行写入和读取，命令如下：

# 写入数据
set name oxyay
set a 1
set b 2
# 读取数据
get name 
get a
get b

192.168.124.3:6374> set name oxyay
OK
192.168.124.3:6374> set a 1
-> Redirected to slot [15495] located at 192.168.124.4:6372
OK
192.168.124.4:6372> set b 2
-> Redirected to slot [3300] located at 192.168.124.4:6371
OK
192.168.124.3:6374> get name
"oxyay"
192.168.124.3:6374> get a
-> Redirected to slot [15495] located at 192.168.124.4:6372
"1"
192.168.124.4:6372> get b
-> Redirected to slot [3300] located at 192.168.124.4:6371
"2"
192.168.124.4:6371>