环境:CentOS7.9
版本:redis-6.2.11
一、下载
http://download.redis.io/releases/
二、说明
redis5.0版本之后可以直接使用redis-cli命令创建集群,不使用redis-trib.rb命令了。
本文在一台主机上建立3主3从集群。实际最少需要3台服务器,6个节点。当然6个服务器更好了。
rdb和aof可以同时开启。如果只做缓存,可以不开启aof。
Redis 集群的数据分片
Redis 集群没有使用一致性hash, 而是引入了 哈希槽的概念.
Redis 集群有16384个哈希槽,每个key通过CRC16校验后对16384取模来决定放置哪个槽.集群的每个节点负责一部分hash槽,举个例子,比如当前集群有3个节点,那么:
- 节点 A 包含 0 到 5500号哈希槽.
- 节点 B 包含5501 到 11000 号哈希槽.
- 节点 C 包含11001 到 16384号哈希槽.
这种结构很容易添加或者删除节点. 比如如果我想新添加个节点D, 我需要从节点 A, B, C中得部分槽到D上. 如果我想移除节点A,需要将A中的槽移到B和C节点上,然后将没有任何槽的A节点从集群中移除即可. 由于从一个节点将哈希槽移动到另一个节点并不会停止服务,所以无论添加删除或者改变某个节点的哈希槽的数量都不会造成集群不可用的状态.
Redis 集群的目标
Redis 集群是 Redis 的一个分布式实现,主要是为了实现以下这些目标(按在设计中的重要性排序):
- 在1000个节点的时候仍能表现得很好并且可扩展性(scalability)是线性的。
- 没有合并操作,这样在 Redis 的数据模型中最典型的大数据值中也能有很好的表现。
- 写入安全(Write safety):那些与大多数节点相连的客户端所做的写入操作,系统尝试全部都保存下来。不过公认的,还是会有小部分(small windows?)写入会丢失。
- 可用性(Availability):在绝大多数的主节点(master node)是可达的,并且对于每一个不可达的主节点都至少有一个它的从节点(slave)可达的情况下,Redis 集群仍能进行分区(partitions)操作。
实现的功能子集
Redis 集群实现了所有在非分布式 Redis 版本中出现的处理单一键值(key)的命令。那些使用多个键值的复杂操作, 比如 set 里的并集(unions)和交集(intersections)操作,就没有实现。通常来说,那些处理命令的节点获取不到键值的所有操作都不会被实现。 在将来,用户或许可以通过使用 MIGRATE COPY 命令,在集群上用 计算节点(Computation Nodes) 来执行多键值的只读操作, 但 Redis 集群本身不会执行复杂的多键值操作来把键值在节点间移来移去。 Redis 集群不像单机版本的 Redis 那样支持多个数据库,集群只有数据库 0,而且也不支持 SELECT 命令。
三、准备工作
主IP | 从复制谁 (不是自己) | 端口 | 集群端口 | 备注 |
---|---|---|---|---|
172.10.10.177 | 172.10.10.179 | 主:6479 从:6579 | 16479 16579 | 主节点 从节点 |
172.10.10.178 | 172.10.10.177 | 主:6479 从:6579 | 16479 16579 | 主节点 从节点 |
172.10.10.179 | 172.10.10.179 | 主:6479 从:6579 | 16479 16579 | 主节点 从节点 |
从最好不是复制的自己,如果不满意可以重建集群
集群总线
每个Redis集群中的节点都需要打开两个TCP连接。一个连接用于正常的给Client提供服务,比如6379,还有一个额外的端口(通过在这个端口号上加10000)作为数据端口,例如:redis的端口为6379,那么另外一个需要开通的端口是:6379 + 10000, 即需要开启 16379。16379端口用于集群总线,这是一个用二进制协议的点对点通信信道。这个集群总线(Cluster bus)用于节点的失败侦测、配置更新、故障转移授权,等等。
防火墙开通端口
firewall-cmd --permanent --zone=public --add-port=6479/tcp
firewall-cmd --permanent --zone=public --add-port=6579/tcp
firewall-cmd --permanent --zone=public --add-port=16479/tcp
firewall-cmd --permanent --zone=public --add-port=16579/tcp
firewall-cmd --reload
创建目录
[root@test2 ~]# cd /opt/redis6
[root@test2 redis6]# mkdir 6479 6579
复制配置文件,并编辑
[root@test2 redis6]# cp /opt/redis6/redis.conf /opt/redis6/6479/redis.conf
[root@test2 redis6]# cp /opt/redis6/redis.conf /opt/redis6/6579/redis.conf
开启集群配置
#启用集群模式
cluster-enabled yes
#集群配置文件(自动创建)
# cluster-config-file nodes-6379.conf
cluster-config-file /opt/redis6/6479/nodes-6479.conf
#cluster-config-file /opt/redis6/6479/nodes-6459.conf
#超时时间
# cluster-node-timeout 15000
cluster-node-timeout 5000
四、安装redis单例(主从)
CentOS7 安装redis6.2,主从复制,配置成服务,自启动,replicaof ,slaveof
创建集群,不用配置主从复制replicaof
五、启动redis
分别启动redis
/opt/redis6/bin/redis-server /opt/redis6/6479/redis.conf
/opt/redis6/bin/redis-server /opt/redis6/6579/redis.conf
六、创建集群
关键的就是这么一句
# --cluster-replicas 1 表示主从配置比,1表示的是1:1,前三个是主,后三个是从
# 若配置文件中设置的密码,则还需要加上-a passwod
ip使用局域网IP,方便远程访问
[root@test2 bin]# /opt/redis6/bin/redis-cli --cluster create 172.10.10.177:6479 172.10.10.178:6479 172.10.10.179:6479 172.10.10.177:6579 172.10.10.178:6579 172.10.10.179:6579 --cluster-replicas 1 -a redis6211
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.10.10.178:6579 to 172.10.10.177:6479
Adding replica 172.10.10.179:6579 to 172.10.10.178:6479
Adding replica 172.10.10.177:6579 to 172.10.10.179:6479
M: 5222ac88da5eb971cd13f71d226f15558315f523 172.10.10.177:6479
slots:[0-5460] (5461 slots) master
M: 4264cb4aa2a7f13525b7f3f05994c0198466997a 172.10.10.178:6479
slots:[5461-10922] (5462 slots) master
M: ecbbd1bb4d50fb77715c54889011c20c36fbb281 172.10.10.179:6479
slots:[10923-16383] (5461 slots) master
S: 8eb78568786c7bcc6c3ecb5bc97127ce89631d1d 172.10.10.177:6579
replicates ecbbd1bb4d50fb77715c54889011c20c36fbb281
S: 5a043cc00d2666a9c8187221ea80300890e216c3 172.10.10.178:6579
replicates 5222ac88da5eb971cd13f71d226f15558315f523
S: 5c1a35f176793111c0d58efbcad576ab9bde50b1 172.10.10.179:6579
replicates 4264cb4aa2a7f13525b7f3f05994c0198466997a
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.10.10.177:6479)
M: 5222ac88da5eb971cd13f71d226f15558315f523 172.10.10.177:6479
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 4264cb4aa2a7f13525b7f3f05994c0198466997a 172.10.10.178:6479
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 8eb78568786c7bcc6c3ecb5bc97127ce89631d1d 172.10.10.177:6579
slots: (0 slots) slave
replicates ecbbd1bb4d50fb77715c54889011c20c36fbb281
S: 5c1a35f176793111c0d58efbcad576ab9bde50b1 172.10.10.179:6579
slots: (0 slots) slave
replicates 4264cb4aa2a7f13525b7f3f05994c0198466997a
S: 5a043cc00d2666a9c8187221ea80300890e216c3 172.10.10.178:6579
slots: (0 slots) slave
replicates 5222ac88da5eb971cd13f71d226f15558315f523
M: ecbbd1bb4d50fb77715c54889011c20c36fbb281 172.10.10.179:6479
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
七、查看日志
tail -f /opt/redis6/6479/redis.log
tail -f /opt/redis6/6579/redis.log
八、手动启停
/opt/redis6/bin/redis-server /opt/redis6/6479/redis.conf
/opt/redis6/bin/redis-server /opt/redis6/6579/redis.conf
/opt/redis6/bin/redis-cli -c -h 127.0.0.1 -p 6479 -a redis6211 shutdown
/opt/redis6/bin/redis-cli -c -h 127.0.0.1 -p 6579 -a redis6211 shutdown
九、连接测试
[root@dev2 ~]# /opt/redis6/bin/redis-cli -c -h 127.0.0.1 -p 6479 -a redis6211
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6479> get foo
-> Redirected to slot [12182] located at 172.10.10.179:6479
(nil)
172.10.10.179:6479> set foo bar
OK
172.10.10.179:6479> get foo
"bar"
172.10.10.179:6479> get foo2
"bar2"
172.10.10.179:6479> get foo3
"bar3"
172.10.10.179:6479> set hello world
-> Redirected to slot [866] located at 172.10.10.177:6479
OK
172.10.10.177:6479> del foo
-> Redirected to slot [12182] located at 172.10.10.179:6479
(integer) 1
172.10.10.179:6479> del hello
-> Redirected to slot [866] located at 172.10.10.177:6479
(integer) 1
172.10.10.177:6479> save
OK
172.10.10.177:6479> exit
十、查看集群状态
[root@test2 ~]# /opt/redis6/bin/redis-cli -h 127.0.0.1 -p 6479 -a redis6211
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6479> cluster nodes
afb78d482884444027f1aef4f54df2bf7da7ccf8 172.10.10.179:6479@16479 master - 0 1681784224480 3 connected 10923-16383
99d21673e03a8b7ab47ad1be28fc2b6ee63f391e 172.10.10.179:6579@16579 slave fe29e4fdf577ee326bb7a66a7669042c718caff3 0 1681784222000 2 connected
244635b346f6190072cd3d9311f52e83ba698f97 172.10.10.178:6579@16579 slave a7eb9bf3ad3817659d3a3cb09543600e95b1b987 0 1681784224000 1 connected
fe29e4fdf577ee326bb7a66a7669042c718caff3 172.10.10.178:6479@16479 master - 0 1681784223000 2 connected 5461-10922
a25a2ff556bc68b96ce55d997aa33ec1ef23ae30 172.10.10.177:6579@16579 slave afb78d482884444027f1aef4f54df2bf7da7ccf8 0 1681784222475 3 connected
a7eb9bf3ad3817659d3a3cb09543600e95b1b987 172.10.10.177:6479@16479 myself,master - 0 1681784223000 1 connected 0-5460
127.0.0.1:6479> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:2989846
cluster_stats_messages_pong_sent:3085402
cluster_stats_messages_sent:6075248
cluster_stats_messages_ping_received:3085397
cluster_stats_messages_pong_received:2989846
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:6075248
127.0.0.1:6479> cluster help
1) CLUSTER <subcommand> [<arg> [value] [opt] ...]. Subcommands are:
2) ADDSLOTS <slot> [<slot> ...]
3) Assign slots to current node.
4) BUMPEPOCH
5) Advance the cluster config epoch.
6) COUNT-FAILURE-REPORTS <node-id>
7) Return number of failure reports for <node-id>.
8) COUNTKEYSINSLOT <slot>
9) Return the number of keys in <slot>.
10) DELSLOTS <slot> [<slot> ...]
11) Delete slots information from current node.
12) FAILOVER [FORCE|TAKEOVER]
13) Promote current replica node to being a master.
14) FORGET <node-id>
15) Remove a node from the cluster.
16) GETKEYSINSLOT <slot> <count>
17) Return key names stored by current node in a slot.
18) FLUSHSLOTS
19) Delete current node own slots information.
20) INFO
21) Return information about the cluster.
22) KEYSLOT <key>
23) Return the hash slot for <key>.
24) MEET <ip> <port> [<bus-port>]
25) Connect nodes into a working cluster.
26) MYID
27) Return the node id.
28) NODES
29) Return cluster configuration seen by node. Output format:
30) <id> <ip:port> <flags> <master> <pings> <pongs> <epoch> <link> <slot> ...
31) REPLICATE <node-id>
32) Configure current node as replica to <node-id>.
33) RESET [HARD|SOFT]
34) Reset current node (default: soft).
35) SET-CONFIG-EPOCH <epoch>
36) Set config epoch of current node.
37) SETSLOT <slot> (IMPORTING|MIGRATING|STABLE|NODE <node-id>)
38) Set slot state.
39) REPLICAS <node-id>
40) Return <node-id> replicas.
41) SAVECONFIG
42) Force saving cluster configuration on disk.
43) SLOTS
44) Return information about slots range mappings. Each range is made of:
45) start, end, master and replicas IP addresses, ports and ids
46) HELP
47) Prints this help.
十一、配置成服务,自启动
每台服务器配两个文件,一个主的,一个从的
[root@bogon ~]# vim /usr/lib/systemd/system/redis6cm.service (redis6479.service)
[Unit]
Description=redis6 cluster master
After=network.target
[Service]
Type=forking
LimitNOFILE=infinity
ExecStart=/opt/redis6/bin/redis-server /opt/redis6/6479/redis.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/opt/redis6/bin/redis-cli -c -h 127.0.0.1 -p 6479 -a redis6211 shutdown
PrivateTmp=true
[Install]
WantedBy=multi-user.target
ExecStop这么写比ExecStop=/bin/kill -s QUIT $MAINPID友好点,停止时会做处理,会有日志。
配置项中的意义可以百度搜索:Systemd Unit文件。
启动停止服务
[root@bogon ~]# systemctl start redis6cm
[root@bogon ~]# systemctl stop redis6cm
配置成自启动
[root@bogon ~]# systemctl enable redis6cm
Created symlink from /etc/systemd/system/multi-user.target.wants/redis6cm.service to /usr/lib/systemd/system/redis6cm.service.
配置文件修改后。
Warning: redis6cm.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[root@bogon ~]# systemctl daemon-reload
[root@bogon ~]# vim /usr/lib/systemd/system/redis6cs.service(redis6579.service)
[Unit]
Description=redis6 cluster slave
After=network.target
[Service]
Type=forking
LimitNOFILE=infinity
ExecStart=/opt/redis6/bin/redis-server /opt/redis6/6579/redis.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/opt/redis6/bin/redis-cli -c -h 127.0.0.1 -p 6579 -a redis6211 shutdown
PrivateTmp=true
[Install]
WantedBy=multi-user.target
相似操作
报错处理:
1、创建集群时卡在Waiting for the cluster to join
[root@dev2 ~]# /opt/redis6/bin/redis-cli --cluster create 172.10.10.177:6479 172.10.10.178:6479 172.10.10.179:6479 172.10.10.177:6579 172.10.10.178:6579 172.10.10.179:6579 --cluster-replicas 1 -a redis6211
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.10.10.178:6579 to 172.10.10.177:6479
Adding replica 172.10.10.179:6579 to 172.10.10.178:6479
Adding replica 172.10.10.177:6579 to 172.10.10.179:6479
M: 2ab77fae2773ea7aa0b6dafe4f6e9d3664506e7a 172.10.10.177:6479
slots:[0-5460] (5461 slots) master
M: 4af61908d0f68741423e8c2b7c26d817e4392a39 172.10.10.178:6479
slots:[5461-10922] (5462 slots) master
M: 0e6f56f4557ece524407ea78a0647bffa335bd64 172.10.10.179:6479
slots:[10923-16383] (5461 slots) master
S: 3e7ae534d0910cf2a489c7728c6f3a565584fb83 172.10.10.177:6579
replicates 0e6f56f4557ece524407ea78a0647bffa335bd64
S: 38d20d507d019c5e954a0db7585b268f9802c85f 172.10.10.178:6579
replicates 2ab77fae2773ea7aa0b6dafe4f6e9d3664506e7a
S: 7319ddbc5cbe74e13c91fd93377c81a580e0d092 172.10.10.179:6579
replicates 4af61908d0f68741423e8c2b7c26d817e4392a39
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
集群总线
每个Redis集群中的节点都需要打开两个TCP连接。一个连接用于正常的给Client提供服务,比如6379,还有一个额外的端口(通过在这个端口号上加10000)作为数据端口,例如:redis的端口为6379,那么另外一个需要开通的端口是:6379 + 10000, 即需要开启 16379。16379端口用于集群总线,这是一个用二进制协议的点对点通信信道。这个集群总线(Cluster bus)用于节点的失败侦测、配置更新、故障转移授权,等等。
防火墙打开对应端口
2、创建集群时报错
[ERR] Node 172.10.10.177:6479 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
https://blog.csdn.net/vtopqx/article/details/50235737
不行就重启redis