快速搭建高可用Redis Cluster主从模式集群的填坑之路

Redis Cluster主从模式

原理

redis cluster 为了保证数据的高可用性,加入了主从模式,一个主节点对应一个或多个从节点主节点提供数据存取从节点则是从主节点拉取数据。
备份,当这个主节点挂掉后,就会有这个从节点选取一个来充当主节点,从而保证集群不会挂掉。

说明

假如集群有A、B、C三个主节点, 如果这3个节点都没有加入从节点,如果B挂掉了,我们就无法访问整个集群了, A和C的slot也无法访问。
所以在集群建立的时候,一定要为每个主节点都添加了从节点,。比如,像集群包含主节点A、B、C, 以及从节点A1、B1、C1, 那么即使B挂掉,系统也可以继续正确工作 ,B1节点替代了B节点,故Redis集群将会选择B1节点,作为新的主节点,集群将会继续正确地提供服务 当B重新开启后,它就会变成B1的从节点。

注意

如果节点B和B1同时挂了,Redis集群就无法继续正确地提供服务了。

最小的redis集群需要3个主节点(即Redis Cluster集群至少需要3个master节点,也就是说至少需要6个节点才能构建Redis cluster集群),一台机器可运行多个redis实例一般使用两台机器,每台启动3个redis实例,即三个主节点,三个从节点)。

很多案例使用单台服务器开6个端口,操作差不多,只是配置基本相对简单点,多台服务器更接近生产环境 。

当集群最开始创建好后,要记住各节点的主从关系(或是创建的时候指定主从关系);若是其中一台机器重启,重启后,需重新将其加入到redis cluster集群中;这就需要将这台机器上的各节点之前的从节点变为主节点(客户端执行slaveof no one),然后再根据新的主节点,添加这台机器的各节点到集群中,添加后变为从节点。

搭建环境

环境 : CentOS7 

redis01 : 172.16.7.11 端口 : 7000、7001、7002

redis02 : 172.16.7.22 端口 : 7003、7004、7005

redis03 : 172.16.7.33 端口 : 7006、7007、7008

关闭SElinux和防火墙

安装redis

首先选择一台服务器,进行安装

初始安装redis01

[root@redis01 ~]# yum install -y gcc g++ make gcc-c++ kernel-devel automake autoconf libtool make wget tcl vim ruby rubygems unzip git
[root@redis01 ~]#  wget http://download.redis.io/releases/redis-4.0.1.tar.gz
[root@redis01 ~]# tar xf redis-4.0.1.tar.gz -C /usr/local/src/
[root@redis01 ~]# cd /usr/local/src/redis-4.0.1/
[root@redis01 redis-4.0.1]# make && make test && make PREFIX=/usr/local/redis install
[root@redis01 redis-4.0.1]# mkdir /usr/local/redis/conf
[root@redis01 redis-4.0.1]# cp *.conf /usr/local/redis/conf/
[root@redis01 redis-4.0.1]# cp /usr/local/src/redis-4.0.1/src/redis-trib.rb /usr/local/redis/bin/
[root@redis01 redis-4.0.1]# ln -s /usr/local/redis/bin/* /usr/local/bin/

进行操作系统基础调优设置

[root@redis01 redis-4.0.1]# echo "* - nofile 10240" >> /etc/security/limits.conf 
[root@redis01 redis-4.0.1]# echo "net.core.somaxconn = 10240" >> /etc/sysctl.conf 
[root@redis01 redis-4.0.1]# echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf 
[root@redis01 redis-4.0.1]# sysctl -p
net.core.somaxconn = 10240
vm.overcommit_memory = 1
[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@redis01 redis-4.0.1]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
[root@redis01 redis-4.0.1]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
[root@redis01 redis-4.0.1]# su -l

 创建节点

创建集群节点目录

[root@redis01 ~]# mkdir /data/redis/redis-cluster -p
[root@redis01 redis-cluster]# cd /data/redis/redis-cluster/
[root@redis01 redis-cluster]# mkdir 7000 7001 7002 

执行下面的脚本,配置各个节点的redis.conf

[root@redis01 redis-cluster]# for i in 0 1 2;
 do 
echo -e "cluster-enabled yes\n\
bind 0.0.0.0\n\
port 700$i\n\
pidfile /data/redis-cluster/700$i/redis.pid\n\
logfile "/data/redis-cluster/700$i/redis.log"\n\
dir /data/redis-cluster/7000/\n\
tcp-backlog 1024\n\
timeout 0\n\
cluster-node-timeout 10100\n\
tcp-keepalive 0\n\
daemonize yes\n\
loglevel notice\n\
databases 16\n\
stop-writes-on-bgsave-error yes\n\
rdbcompression yes\n\
rdbchecksum yes\n\
dbfilename "dump.rdb"\n\
slave-serve-stale-data yes\n\
slave-read-only yes\n\
repl-diskless-sync no\n\
repl-diskless-sync-delay 5\n\
repl-disable-tcp-nodelay no\n\
slave-priority 100\n\
lazyfree-lazy-eviction no\n\
lazyfree-lazy-expire no\n\
lazyfree-lazy-server-del no\n\
slave-lazy-flush no\n\
appendonly no\n\
appendfilename "appendonly.aof"\n\
appendfsync everysec\n\
no-appendfsync-on-rewrite yes\n\
auto-aof-rewrite-percentage 100\n\
auto-aof-rewrite-min-size 64mb\n\
aof-load-truncated yes\n\
lua-time-limit 5000\n\
slowlog-log-slower-than 10000\n\
slowlog-max-len 128\n\
latency-monitor-threshold 0\n\
notify-keyspace-events ""\n\
hash-max-ziplist-entries 512\n\
hash-max-ziplist-value 64\n\
list-max-ziplist-entries 512\n\
list-max-ziplist-value 64\n\
set-max-intset-entries 512\n\
zset-max-ziplist-entries 128\n\
zset-max-ziplist-value 64\n\
hll-sparse-max-bytes 3000\n\
activerehashing yes\n\
client-output-buffer-limit normal 0 0 0\n\
client-output-buffer-limit slave 256mb 64mb 60\n\
client-output-buffer-limit pubsub 32mb 8mb 60\n\
hz 10\n\
aof-rewrite-incremental-fsync yes" > 700$i/redis.conf;
 done

 查看各个节点的文件树

[root@redis01 redis-cluster]# tree 
.
├── 7000
│   └── redis.conf
├── 7001
│   └── redis.conf
└── 7002
    └── redis.conf

启动redis01的集群服务

[root@redis01 redis-cluster]for i in 0 1 2;
do redis-server /data/redis/redis-cluster/700$i/redis.conf 
done
68820:C 08 Oct 22:21:14.965 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
68820:C 08 Oct 22:21:14.965 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=68820, just started
68820:C 08 Oct 22:21:14.965 # Configuration loaded
68822:C 08 Oct 22:21:14.972 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
68822:C 08 Oct 22:21:14.972 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=68822, just started
68822:C 08 Oct 22:21:14.972 # Configuration loaded
68824:C 08 Oct 22:21:14.981 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
68824:C 08 Oct 22:21:14.981 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=68824, just started
68824:C 08 Oct 22:21:14.981 # Configuration loaded

redis02 服务器的安装

redis02 ,redis03不需要完全按上面的步骤,只部分需要复制redis01的安装解压目录

复制redis01的安装解压目录

[root@node02 ~]# scp -r node01:/usr/local/redis /usr/local/

[root@redis02 ~]# scp -r node01:/data/redis/redis-cluster /data/redis/

修改某个服务器的各个节点的redis.conf文件

[root@redis02 redis-cluster]# mv 7000 7003
[root@redis02 redis-cluster]# mv 7001 7004
[root@redis02 redis-cluster]# mv 7002 7005
[root@redis02 redis-cluster]# ls
7003  7004  7005
[root@redis02 redis-cluster]# sed -i 's#7000#7003#g' 7003/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7001#7004#g' 7005/redis.conf
[root@redis02 redis-cluster]# sed -i 's#7002#7005#g' 7004/redis.conf

做运行脚本加入bin目录 

[root@node02 ~]# ln -s /usr/local/redis/bin/* /usr/local/bin/

进行操作系统基础调优设置

如同redis01的操作系统基础调优设置

启动redis02的集群服务

[root@redis02 redis-cluster]# for i in 3 4 5;
 do redis-server /data/redis/redis-cluster/700$i/redis.conf 
 done
41766:C 08 Oct 22:21:42.369 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41766:C 08 Oct 22:21:42.369 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41766, just started
41766:C 08 Oct 22:21:42.369 # Configuration loaded
41768:C 08 Oct 22:21:42.375 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41768:C 08 Oct 22:21:42.375 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41768, just started
41768:C 08 Oct 22:21:42.375 # Configuration loaded
41770:C 08 Oct 22:21:42.380 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
41770:C 08 Oct 22:21:42.380 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=41770, just started
41770:C 08 Oct 22:21:42.380 # Configuration loaded

redis03服务器的安装

类比redis02 服务器的安装

redis01,redis02,redis03聚合创建集群

查各Redis节点的启动状态

#yum -y install net-tools
[root@redis01 redis-cluster]# netstat -antup|grep redis
tcp        0      0 172.16.7.11:17000     0.0.0.0:*               LISTEN      68821/redis-server  
tcp        0      0 172.16.7.11:17001     0.0.0.0:*               LISTEN      68823/redis-server  
tcp        0      0 172.16.7.11:17002     0.0.0.0:*               LISTEN      68828/redis-server  
tcp        0      0 172.16.7.11:7000      0.0.0.0:*               LISTEN      68821/redis-server  
tcp        0      0 172.16.7.11:7001      0.0.0.0:*               LISTEN      68823/redis-server  
tcp        0      0 172.16.7.11:7002      0.0.0.0:*               LISTEN      68828/redis-server 

安装ruby

[root@redis01 redis-cluster]# yum -y install rh-ruby23
[root@redis01 redis-cluster]# ruby -v
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
[root@redis01 redis-cluster]# scl enable rh-ruby23 bash
[root@redis01 redis-cluster]# ruby -v
ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]

解决方案;

yum install centos-release-scl-rh

再次安装 rh-ruby23

安装ruby的redis扩展

[root@redis01 redis-cluster]# gem install redis
Fetching: redis-4.0.2.gem (100%)
Successfully installed redis-4.0.2
Parsing documentation for redis-4.0.2
Installing ri documentation for redis-4.0.2
Done installing documentation for redis after 0 seconds
1 gem installed

使用redis自带的集群管理工具redis-trib.rb创建集群

在redis01上运行以下脚本

[root@redis01 redis-cluster]# redis-trib.rb create --replicas 1 172.16.7.11:7000 172.16.7.11:7001 172.16.7.11:7002 172.16.7.22:7003 172.16.7.22:7004 172.16.7.22:7005 172.16.7.33:7006 172.16.7.33:7007 172.16.7.33:7008
>>> Creating cluster
>>> Performing hash slots allocation on 9 nodes...
Using 4 masters:
172.16.7.11:7000
172.16.7.22:7003
172.16.7.33:7006
172.16.7.11:7001
Adding replica 172.16.7.22:7004 to 172.16.7.11:7000
Adding replica 172.16.7.33:7007 to 172.16.7.22:7003
Adding replica 172.16.7.11:7002 to 172.16.7.33:7006
Adding replica 172.16.7.22:7005 to 172.16.7.11:7001
Adding replica 172.16.7.33:7008 to 172.16.7.11:7000
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
   slots:0-4095 (4096 slots) master
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
   slots:12288-16383 (4096 slots) master
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
   replicates af2e1c600ed2511155a6cad61cabeb095affff86
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
   slots:4096-8191 (4096 slots) master
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
   replicates d8138e15d9cb69962abb7457125a7965b735a658
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
   slots:8192-12287 (4096 slots) master
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
   replicates d8138e15d9cb69962abb7457125a7965b735a658
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........
>>> Performing Cluster Check (using node 172.16.7.11:7000)
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
   slots:0-4095 (4096 slots) master
   2 additional replica(s)
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
   slots: (0 slots) slave
   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
   slots:4096-8191 (4096 slots) master
   1 additional replica(s)
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
   slots:8192-12287 (4096 slots) master
   1 additional replica(s)
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
   slots: (0 slots) slave
   replicates af2e1c600ed2511155a6cad61cabeb095affff86
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
   slots: (0 slots) slave
   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

这样,三个服务器启动了9个实例,配置成4主5从,其中一个是一主两从,其他都是一主一从

集群验证

选项 -c 可连接集群, 因为配置文件中bind改成了ip,所以必须要用 -h 选项,p选项后接端口号

在redis01上的7000端口的实例上set一个key

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000
172.16.7.11:7000> set name haha
-> Redirected to slot [5798] located at 172.16.7.22:7003
OK
172.16.7.22:7003> get name
"haha"

由上可见 , redis01 set name 后,将数据重定向到了redis02的7003端口的实例上

在redis03的7008实例上get一个key

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.33 -c -p 7008
172.16.7.33:7008> get name
-> Redirected to slot [5798] located at 172.16.7.22:7003
"weihan"
172.16.7.22:7003>

 get name 还是会重定向到7003这个实例上 , 这说明redis cluster集群已经是可用的了

redis-trib.rb检查集群状态

4个master会分配到4个服务器上, 而且每个服务器上必须有一个master,创建集群的时候,可以指定master,若创建的时候没指定,是默认分配

[root@redis01 redis-cluster]# redis-trib.rb check 172.16.7.11:7000
>>> Performing Cluster Check (using node 172.16.7.11:7000)
M: d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000
   slots:0-4095 (4096 slots) master
   2 additional replica(s)
S: b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007
   slots: (0 slots) slave
   replicates 691428ae7c1728a76520b8f9d46fa6d83930f191
S: e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
M: 691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003
   slots:4096-8191 (4096 slots) master
   1 additional replica(s)
M: af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006
   slots:8192-12287 (4096 slots) master
   1 additional replica(s)
M: 72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002
   slots: (0 slots) slave
   replicates af2e1c600ed2511155a6cad61cabeb095affff86
S: ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005
   slots: (0 slots) slave
   replicates 72f44da597f73c8d23c6d3fb60fa95489eb0168d
S: 6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008
   slots: (0 slots) slave
   replicates d8138e15d9cb69962abb7457125a7965b735a658
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

列出集群节点

[root@redis01 redis-cluster]# redis-cli -h 172.16.7.11 -c -p 7000
172.16.7.11:7000> CLUSTER NODES
b1e201b2acdc0a0740a764978436fa3946325a11 172.16.7.33:7007@17007 slave 691428ae7c1728a76520b8f9d46fa6d83930f191 0 1539016061737 8 connected
e752d865374c8ca4149737d81f19ac95cc9f6a44 172.16.7.22:7004@17004 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016062000 5 connected
691428ae7c1728a76520b8f9d46fa6d83930f191 172.16.7.22:7003@17003 master - 0 1539016064655 4 connected 4096-8191
af2e1c600ed2511155a6cad61cabeb095affff86 172.16.7.33:7006@17006 master - 0 1539016062000 7 connected 8192-12287
72f44da597f73c8d23c6d3fb60fa95489eb0168d 172.16.7.11:7001@17001 master - 0 1539016063648 2 connected 12288-16383
711af78396c3ba966f751beef745ae10ff053a3f 172.16.7.11:7002@17002 slave af2e1c600ed2511155a6cad61cabeb095affff86 0 1539016063000 7 connected
d8138e15d9cb69962abb7457125a7965b735a658 172.16.7.11:7000@17000 myself,master - 0 1539016063000 1 connected 0-4095
ba4ce74ba7b84bde9ea60cd8bf4c5996f1ca01e5 172.16.7.22:7005@17005 slave 72f44da597f73c8d23c6d3fb60fa95489eb0168d 0 1539016063000 6 connected
6913f44486938c49eed6efa6f18c782df73e6456 172.16.7.33:7008@17008 slave d8138e15d9cb69962abb7457125a7965b735a658 0 1539016063000 9 connected

打印集群信息

172.16.7.11:7000> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:4
cluster_current_epoch:9
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1240
cluster_stats_messages_pong_sent:1251
cluster_stats_messages_sent:2491
cluster_stats_messages_ping_received:1243
cluster_stats_messages_pong_received:1240
cluster_stats_messages_meet_received:8
cluster_stats_messages_received:2491

备注

若指定redis cluster中的master和slave ,可以用下面的方法

# 添加主库
[root@redis01 redis-cluster]# redis-trib.rb create 172.16.7.11:7000 172.16.7.22:7003 172.16.7.33:7006
# 添加从库
[root@redis01 redis-cluster]# redis-trib.rb add-node --slave 172.16.7.22:7001 172.16.7.11:7000    # 把172.16.7.11的7001 作为从库添加到172.16.7.11的7000

 

©️2020 CSDN 皮肤主题: 精致技术 设计师:CSDN官方博客 返回首页