docker redis 6.2 cluster (一) 随机分配三主三从

ppdouble

已于 2022-05-19 23:51:27 修改

阅读量782

点赞数

分类专栏： Java 文章标签： docker redis 容器

于 2022-05-08 09:04:07 首次发布

本文链接：https://blog.csdn.net/ppdouble/article/details/124567634

版权

Java 专栏收录该内容

22 篇文章 1 订阅

订阅专栏

文章目录

[docker-compose redis 6.2 cluster (二) 随机分配三主三从]

ENV

Docker version 19.03.5, build 633a0ea838
redis 6.2

code

code on github

1 单机

1.1 创建容器

创建容器，并设置 volume 映射

docker run --publish 6379:6379 --publish 16379:16379 --name redis --volume $(pwd)/data:/data redis:6.2

--publish=6379:6379 --publish=16379:16379: 端口映射
--name redis: 容器名
--volume=$(pwd)/data:/data: volume 映射
redis:6.2: 指定所使用的 image

1.2 进入容器

进入容器，并在映射 volume 下创建文件

[hostuser@host-machine]$ docker exec -it redis bash
root@eddf2c42677b:/data# ls -l
total 0
root@eddf2c42677b:/data# echo "root in container" > r.txt
root@eddf2c42677b:/data# su redis
$ whoami
redis
$ id redis
uid=999(redis) gid=999(redis) groups=999(redis)
$ id root
uid=0(root) gid=0(root) groups=0(root)
$ echo "user in container" > u.txt
$ ls -l
total 8
-rw-r--r--. 1 root  root  18 May  4 04:51 r.txt
-rw-r--r--. 1 redis redis 18 May  4 04:51 u.txt
$ pwd
/data
$

reddis:6.2 镜像里配置了用户名 redis。可以看到用户名 root 和 redis 在container 里的 ID 分别为 0，999。
从 host 里的映射 volume 里看:

drwxr-xr-x. 2 100999 100000 4096 May  4 12:51 data

ls -l data
total 8
-rw-r--r--. 1 100000 100000 18 May  4 12:51 r.txt
-rw-r--r--. 1 100999 100999 18 May  4 12:51 u.txt

映射 volume 目录 data 也是 container 的用户创建的，其 owner 100999:100000 对应的是 container 里的用户是 999:0 即 redis:root。所以 100000 对应是 container 里的 root 用户，100999 对应的是 container 里的 redis 用户。

1.2.1 创建用户(不建议用于生产环境)

用上面的 id 在host上创建用户

[hostuser@host-machine]$ sudo groupadd -g 100000 redis-docker-root
[hostuser@host-machine]$ sudo groupadd -g 100999 redis-docker-user
[hostuser@host-machine]$ sudo useradd -u 100000 -g redis-docker-root redis-docker-root
[hostuser@host-machine]$ sudo useradd -u 100999 -g redis-docker-user redis-docker-user

这样再从 host 里的映射 volume 里看:

drwxrwxr-x. 2 redis-docker-user redis-docker-root 4096 May  4 14:39 data

ls -l data
total 8
-rw-r--r--. 1 redis-docker-root redis-docker-root 18 May  4 14:33 r.txt
-rw-r--r--. 1 redis-docker-user redis-docker-user 27 May  4 14:35 u.txt

2 创建集群

2.1 创建配置文件和用于volume映射的权限

basepath="/home/data/cluster"
docker network create redis-net --subnet 172.28.0.0/16
for port in $(seq 1 6); \
do \
mkdir -p ./node-${port}/conf
mkdir -p ./node-${port}/data
touch ./node-${port}/conf/redis.conf
cat << EOF > ./node-${port}/conf/redis.conf
#端口 
port 6379
#非保护模式 
protected-mode no
#启用集群模式 
cluster-enabled yes
cluster-config-file nodes.conf
#超时时间 
cluster-node-timeout 15000
#集群各节点IP地址，记得修改为你的ip地址
cluster-announce-ip 172.28.0.1${port}
#集群节点映射端口 
cluster-announce-port 6379
#集群总线端口 
cluster-announce-bus-port 16379 
#开启aof持久化策略 
appendonly yes
#后台运行
#daemonize yes
#进程号存储
pidfile /var/run/redis_6379.pid
#外部访问
bind 0.0.0.0
#集群加密
#masterauth itheima
#requirepass itheima
EOF
cd ${basepath}/redis/
sudo chown -R redis-docker-root:redis-docker-root ${basepath}/redis/node-${port}
sudo chown -R redis-docker-user:redis-docker-root ${basepath}/redis/node-${port}/data
done

2.1.1目录结构

[hostuser@host-machine]$ ls -l .
drwxrwxr-x. 4 redis-docker-root redis-docker-root 4096 May  8 00:43 node-1
drwxrwxr-x. 4 redis-docker-root redis-docker-root 4096 May  8 00:43 node-2
drwxrwxr-x. 4 redis-docker-root redis-docker-root 4096 May  8 00:43 node-3
drwxrwxr-x. 4 redis-docker-root redis-docker-root 4096 May  8 00:43 node-4
drwxrwxr-x. 4 redis-docker-root redis-docker-root 4096 May  8 00:43 node-5
drwxrwxr-x. 4 redis-docker-root redis-docker-root 4096 May  8 00:43 node-6
[hostuser@host-machine]$ ls -l node-1/
total 8
drwxrwxr-x. 2 redis-docker-root redis-docker-root 4096 May  8 00:43 conf
drwxrwxr-x. 2 redis-docker-user redis-docker-root 4096 May  8 00:54 data
[hostuser@host-machine]$ ls -l node-1/conf
total 4
-rw-rw-r--. 1 redis-docker-root redis-docker-root 568 May  8 00:43 redis.conf

.
├── node-1
│   ├── conf
│   │   └── redis.conf
│   └── data
├── node-2
│   ├── conf
│   │   └── redis.conf
│   └── data
├── node-3
│   ├── conf
│   │   └── redis.conf
│   └── data
├── node-4
│   ├── conf
│   │   └── redis.conf
│   └── data
├── node-5
│   ├── conf
│   │   └── redis.conf
│   └── data
└── node-6
    ├── conf
    │   └── redis.conf
    └── data

2.2 创建容器作为集群结点

for port in $(seq 1 6); \
do \
    docker run --publish 637${port}:6379 --publish 1637${port}:16379 --name redis-node-${port} \
        --volume ${basepath}/redis/node-${port}/data:/data \
        --volume ${basepath}/redis/node-${port}/conf/redis.conf:/etc/redis/redis.conf \
        --detach --net redis-net --ip 172.28.0.1${port} redis:6.2 redis-server /etc/redis/redis.conf
done

docker ps -a | grep redis-node
#在NetworkSettings的Networks下的redis-net中查找ip
for port in `seq 1 6`;
do
	echo -n "$(docker inspect --format '{{ (index .NetworkSettings.Networks "redis-net").IPAddress }}' "redis-node-${port}")":637${port}" ";
done

也可以不创建 data 目录，由容器里的用户自动创建。如果手动创建 data 目录，也需要手动修改属主，否则容器里的用户没有权限。

2.2.1 从host进入其中一个结点

[hostuser@host-machine]$ docker exec --interactive --tty redis-node-1 bash
root@63513cddea37:/data# ls
appendonly.aof	nodes.conf
root@63513cddea37:/data#

此时host的目录为

[hostuser@host-machine]$ ls -l node-1/data/
total 8
-rw-r--r--. 1 redis-docker-user redis-docker-user   0 May  8 00:44 appendonly.aof
-rw-r--r--. 1 redis-docker-user redis-docker-user 175 May  8 00:54 dump.rdb
-rw-r--r--. 1 redis-docker-user redis-docker-user 793 May  8 00:54 nodes.conf
[hostuser@host-machine]$ ls -l node-1/conf/
total 4
-rw-rw-r--. 1 redis-docker-root redis-docker-root 568 May  8 00:43 redis.conf

.
├── node-1
│   ├── conf
│   │   └── redis.conf
│   └── data
│       ├── appendonly.aof
│       ├── dump.rdb
│       └── nodes.conf
├── node-2
│   ├── conf
│   │   └── redis.conf
│   └── data
│       ├── appendonly.aof
│       ├── dump.rdb
│       └── nodes.conf
├── node-3
│   ├── conf
│   │   └── redis.conf
│   └── data
│       ├── appendonly.aof
│       ├── dump.rdb
│       └── nodes.conf
├── node-4
│   ├── conf
│   │   └── redis.conf
│   └── data
│       ├── appendonly.aof
│       ├── dump.rdb
│       └── nodes.conf
├── node-5
│   ├── conf
│   │   └── redis.conf
│   └── data
│       ├── appendonly.aof
│       ├── dump.rdb
│       └── nodes.conf
└── node-6
    ├── conf
    │   └── redis.conf
    └── data
        ├── appendonly.aof
        ├── dump.rdb
        └── nodes.conf

2.3 创建集群

在该结点用，使用 redis-cli 创建集群，将六个结点随机分配为三主三从

root@63513cddea37:/data# redis-cli --cluster create 172.28.0.11:6379 172.28.0.12:6379 172.28.0.13:6379 172.28.0.14:6379 172.28.0.15:6379 172.28.0.16:6379 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.28.0.15:6379 to 172.28.0.11:6379
Adding replica 172.28.0.16:6379 to 172.28.0.12:6379
Adding replica 172.28.0.14:6379 to 172.28.0.13:6379
M: ce99907711e032bb13c9c26b9c31369808a80ad3 172.28.0.11:6379
   slots:[0-5460] (5461 slots) master
M: 1d8911fb5b06e4888fac75c6ab7a5af917cea172 172.28.0.12:6379
   slots:[5461-10922] (5462 slots) master
M: b2e81ba1c006f31751b34ecda4f0942fe3e7a679 172.28.0.13:6379
   slots:[10923-16383] (5461 slots) master
S: 731b8b2792d81519f8cb7b8cdb4fb746404ecf75 172.28.0.14:6379
   replicates b2e81ba1c006f31751b34ecda4f0942fe3e7a679
S: a85a113440e81aa47690a95f79f2f27317571ff5 172.28.0.15:6379
   replicates ce99907711e032bb13c9c26b9c31369808a80ad3
S: bd98d6d6684fed834d6cf9a57d50a1a39ad74493 172.28.0.16:6379
   replicates 1d8911fb5b06e4888fac75c6ab7a5af917cea172
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.28.0.11:6379)
M: ce99907711e032bb13c9c26b9c31369808a80ad3 172.28.0.11:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 731b8b2792d81519f8cb7b8cdb4fb746404ecf75 172.28.0.14:6379
   slots: (0 slots) slave
   replicates b2e81ba1c006f31751b34ecda4f0942fe3e7a679
M: 1d8911fb5b06e4888fac75c6ab7a5af917cea172 172.28.0.12:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: b2e81ba1c006f31751b34ecda4f0942fe3e7a679 172.28.0.13:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: a85a113440e81aa47690a95f79f2f27317571ff5 172.28.0.15:6379
   slots: (0 slots) slave
   replicates ce99907711e032bb13c9c26b9c31369808a80ad3
S: bd98d6d6684fed834d6cf9a57d50a1a39ad74493 172.28.0.16:6379
   slots: (0 slots) slave
   replicates 1d8911fb5b06e4888fac75c6ab7a5af917cea172
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@63513cddea37:/data#

可以看到结点的从属关系， M 为master结点， S 为 slave 结点

2.3 使用 Jedis 进行连接，随便添加一个结点

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;

import java.util.HashSet;
import java.util.Set;

public class Main {
    public static void main (String[] args) {
        Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
        jedisClusterNodes.add(new HostAndPort("localhost", 6373));
        JedisCluster jedis = new JedisCluster(jedisClusterNodes);
        System.out.println(jedis.getClusterNodes().keySet());
        jedis.set("planets", "Mars");
        System.out.println(jedis.get("planets"));
    }
}

在字符界面获取 java jedis 添加的数据，在某一个集群结点container里运行 redis-cli -c 进入集群

[hostuser@host-machine]$ docker exec --interactive --tty redis-node-6 bash
root@e147a1e822a1:/data# redis-cli -c
127.0.0.1:6379> get planets
-> Redirected to slot [8813] located at 172.28.0.12:6379
"Mars"
172.28.0.12:6379>

[hostuser@host-machine]$ docker exec --interactive --tty redis-node-6 bash
root@e147a1e822a1:/data# redis-cli -h 172.28.0.14 -c
172.28.0.14:6379> get planets
-> Redirected to slot [8813] located at 172.28.0.12:6379
"Mars"
172.28.0.12:6379>

2.4 关闭主结点

11 主 - 15 从, 12 主 - 16 从, 13 主 - 14 从。

[codc@192 tmp4]$ docker exec -it redis-node-6 bash
root@e147a1e822a1:/data# redis-cli -c
127.0.0.1:6379> cluster nodes
a85a113440e81aa47690a95f79f2f27317571ff5 172.28.0.15:6379@16379 slave ce99907711e032bb13c9c26b9c31369808a80ad3 0 1651999725081 1 connected
b2e81ba1c006f31751b34ecda4f0942fe3e7a679 172.28.0.13:6379@16379 master - 0 1651999723072 3 connected 10923-16383
bd98d6d6684fed834d6cf9a57d50a1a39ad74493 172.28.0.16:6379@16379 myself,slave 1d8911fb5b06e4888fac75c6ab7a5af917cea172 0 1651999720000 2 connected
ce99907711e032bb13c9c26b9c31369808a80ad3 172.28.0.11:6379@16379 master - 0 1651999723000 1 connected 0-5460
1d8911fb5b06e4888fac75c6ab7a5af917cea172 172.28.0.12:6379@16379 master - 0 1651999721063 2 connected 5461-10922
731b8b2792d81519f8cb7b8cdb4fb746404ecf75 172.28.0.14:6379@16379 slave b2e81ba1c006f31751b34ecda4f0942fe3e7a679 0 1651999724076 3 connected
127.0.0.1:6379> GET planets
-> Redirected to slot [8813] located at 172.28.0.12:6379
"Mars"
127.0.0.1:6379> quit
root@e147a1e822a1:/data# redis-cli -c
127.0.0.1:6379> cluster nodes
a85a113440e81aa47690a95f79f2f27317571ff5 172.28.0.15:6379@16379 slave ce99907711e032bb13c9c26b9c31369808a80ad3 0 1651999942053 1 connected
b2e81ba1c006f31751b34ecda4f0942fe3e7a679 172.28.0.13:6379@16379 master,fail - 1651999773287 1651999770000 3 connected
bd98d6d6684fed834d6cf9a57d50a1a39ad74493 172.28.0.16:6379@16379 myself,master - 0 1651999941000 8 connected 5461-10922
ce99907711e032bb13c9c26b9c31369808a80ad3 172.28.0.11:6379@16379 master - 0 1651999940045 1 connected 0-5460
1d8911fb5b06e4888fac75c6ab7a5af917cea172 172.28.0.12:6379@16379 master,fail - 1651999875800 1651999872000 2 connected
731b8b2792d81519f8cb7b8cdb4fb746404ecf75 172.28.0.14:6379@16379 master - 0 1651999940000 7 connected 10923-16383
127.0.0.1:6379> quit
root@e147a1e822a1:/data# exit
exit
[codc@192 tmp4]$ docker exec -it redis-node-6 bash
root@e147a1e822a1:/data# redis-cli -h 172.28.0.15 -c
172.28.0.15:6379> GET planets
-> Redirected to slot [8813] located at 172.28.0.16:6379
"Mars"
172.28.0.16:6379>