本模板用于创建一个RedisHA系统,基本信息如下:
代码位置
https://github.com/double12gzh/redis_HA-keepalived-sentinel-reids
节点个数
master node1个, slave node >= 2个
版本信息
redis 4.0.2,ruby 2.4.2,keepalived1.2.13,tcl 8.6.7,Centos 7.0.1406
功能介绍
结构: redis主从+Sentinel+Keepalived。集群创建完成后,会包含一个master节点和多个slave节点,并且Keepalived会在master节点上绑定一个vip。
当master节点挂掉后,由sentinel进程自动选一个slave并将它变为master,同时Keepalived会把vip挂到新的master上。
当原来的master节点恢复后,它会自动变为slave节点。
适应范围
本HA Redis高可用系统,可以保证单个redis实例的高可用,适用于业务量比较小的应用。
集群拓扑
验证方法
1. 查看各个节点上的redis-server和redis-sentinel进程是否正常启动
[root@redis-ha-redis-master-rj72p3anod5f~]# ps aux | grep redis
root 1797 0.2 0.2 147288 9764 ? Ssl 20:50 0:00 /usr/local/bin/redis-server0.0.0.0:6379
root 1803 0.2 0.1 145240 7764 ? Ssl 20:50 0:00 redis-sentinel *:26379 [sentinel]
root 2740 0.0 0.0 112640 976 pts/0 S+ 20:54 0:00 grep --color=auto redis
[root@redis-ha-redis-master-rj72p3anod5f~]#
2. 查看节点的角色(master还是slave。此处使用的是默认端口6379,如果使用其它端口并在/etc/redis/redis.conf中加了bind YOUR_BIND_IP,请执行:redis-cli –p YOUR_PORT –h YOUR_BIND_IP)。
Master 节点。从这里我们可以看到它有两个slave节点,分别是slave0和slave1。
[root@redis-ha-redis-master-rj72p3anod5f~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>info replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.16.13.188,port=6379,state=online,offset=64753,lag=1
slave1:ip=172.16.13.187,port=6379,state=online,offset=64753,lag=1
master_replid:1736f8cd46c1eb1f78c4cdef3130f6aa78b3863f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:64753
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:64753
0.0.0.0:6379>
Slave节点。这里可以看到它所属的master节点是172.16.13.186。
[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-0-merb6ycj2llg~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>info replication
# Replication
role:slave
master_host:172.16.13.186
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:87749
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:1736f8cd46c1eb1f78c4cdef3130f6aa78b3863f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:87749
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:87749
0.0.0.0:6379>
查看sentinel是否正常启动redis-cli –p 26379 info sentinel
3. 进行读写测试。
在master节点上执行如下写命令
[root@redis-ha-redis-master-rj72p3anod5f~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>set name "Jeffrey Guan"
OK
0.0.0.0:6379>
在slave节点上执行如下读命令
[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-0-merb6ycj2llg~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>get name
"JeffreyGuan"
0.0.0.0:6379>
4. 测试主从切换
切换前的基本拓扑如图:
先看一下master节点上的IP信息,应该可以看到eth0有两个IP,其中172.16.13.185这个是VIP,由keepalive管理。
Master节点:
[root@redis-ha-redis-master-rj72p3anod5f~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
link/ether fa:16:3e:ea:e0:64 brdff:ff:ff:ff:ff:ff
inet 172.16.13.186/24 brd 172.16.13.255scope global dynamic eth0
valid_lft 85813sec preferred_lft85813sec
inet 172.16.13.185/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:feea:e064/64 scopelink
valid_lft forever preferred_lft forever
Slave节点:
[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-0-merb6ycj2llg~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
link/ether fa:16:3e:2e:4b:45 brdff:ff:ff:ff:ff:ff
inet 172.16.13.187/24 brd 172.16.13.255scope global dynamic eth0
valid_lft 85824sec preferred_lft85824sec
inet6 fe80::f816:3eff:fe2e:4b45/64 scopelink
valid_lft forever preferred_lft forever
执行以下命令将master节点上的redis-server关掉,然后查看master节点上的IP信息,发现eth0的172.16.13.185这个IP已不存在了。
[root@redis-ha-redis-master-rj72p3anod5f~]# ps aux | grep redis
root 1797 0.1 0.2 147288 9764 ? Ssl 20:50 0:01 /usr/local/bin/redis-server 0.0.0.0:6379
root 1803 0.2 0.1 145240 7764 ? Ssl 20:50 0:01 redis-sentinel *:26379 [sentinel]
root 4554 0.0 0.0 112640 976 pts/0 S+ 21:01 0:00 grep --color=auto redis
[root@redis-ha-redis-master-rj72p3anod5f~]# kill -9 1797
[root@redis-ha-redis-master-rj72p3anod5f~]# ps aux | grep redis
root 1803 0.2 0.1 145240 7764 ? Ssl 20:50 0:01 redis-sentinel *:26379 [sentinel]
root 4613 0.0 0.0 112640 980 pts/0 S+ 21:01 0:00 grep --color=auto redis
[root@redis-ha-redis-master-rj72p3anod5f~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
link/ether fa:16:3e:ea:e0:64 brd ff:ff:ff:ff:ff:ff
inet 172.16.13.186/24 brd 172.16.13.255scope global dynamic eth0
valid_lft 85693sec preferred_lft85693sec
inet6 fe80::f816:3eff:feea:e064/64 scopelink
valid_lft forever preferred_lft forever
[root@redis-ha-redis-master-rj72p3anod5f~]#
查看slave节点上的IP信息,发现172.16.13.185这个IP已漂到了此slave节点上,同时发现此slave节点已变成了master的角色。其中,角色的变换是由redis-sentinel控制,vip的漂移由keepalived控制。
[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-1-ftdajerozp4z~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP>mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:68:07:0c brdff:ff:ff:ff:ff:ff
inet 172.16.13.188/24 brd 172.16.13.255scope global dynamic eth0
valid_lft 85422sec preferred_lft85422sec
inet 172.16.13.185/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe68:70c/64 scopelink
valid_lft forever preferred_lft forever
You have newmail in /var/spool/mail/root
[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-1-ftdajerozp4z~]#
再到原先的master节点上把redis-server启动起来,查看其IP信息及角色,发现它没有了VIP,并且角色变为了slave。
[root@redis-ha-redis-master-rj72p3anod5f~]# redis-cli info replication
# Replication
role:slave
master_host:172.16.13.188
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:210910
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:67b479dee278eddd337e6a40ecfa4965861b32b5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:210910
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:135372
repl_backlog_histlen:75539
[root@redis-ha-redis-master-rj72p3anod5f~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
link/ether fa:16:3e:ea:e0:64 brdff:ff:ff:ff:ff:ff
inet 172.16.13.186/24 brd 172.16.13.255scope global dynamic eth0
valid_lft 85281sec preferred_lft 85281sec
inet6 fe80::f816:3eff:feea:e064/64 scopelink
valid_lft forever preferred_lft forever
[root@redis-ha-redis-master-rj72p3anod5f~]#
切换后的拓扑如图:
关于VIP漂移,可以查看log:/var/log/message
Master节点上,可以看到VIP被移除了。
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1836]: VRRP_Group(VG_1) Syncinginstances to FAULT state
Sep 30 21:02:26base-server-with-network avahi-daemon[397]: Withdrawing address record for172.16.13.185 on eth0.
Sep 30 21:02:26base-server-with-network Keepalived_healthcheckers[1835]: Netlink reflectorreports IP 172.16.13.185 removed
Slave节点上,可以看到VIP被添加了。
Sep 30 21:02:13base-server-with-network Keepalived_healthcheckers[1837]: Netlink reflectorreports IP 172.16.13.185 removed
Sep 30 21:02:13base-server-with-network avahi-daemon[430]: Withdrawing address record for172.16.13.185 on eth0.
Sep 30 21:02:13base-server-with-network Keepalived_vrrp[1838]: SMTP alert successfully sent.
Sep 30 21:02:25base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) Transitionto MASTER STATE
Sep 30 21:02:25base-server-with-network Keepalived_vrrp[1838]: VRRP_Group(VG_1) Syncinginstances to MASTER state
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) EnteringMASTER STATE
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) settingprotocol VIPs.
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 172.16.13.185
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: Remote SMTP server[127.0.0.1]:25 connected.
Sep 30 21:02:26base-server-with-network Keepalived_healthcheckers[1837]: Netlink reflectorreports IP 172.16.13.185 added
Sep 30 21:02:26base-server-with-network avahi-daemon[430]: Registering new address record for172.16.13.185 on eth0.IPv4.
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: SMTP alert successfully sent.
Sep 30 21:02:31base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 172.16.13.185