openstack heat redis_HA_redis_sentinel_keepalived

本模板用于创建一个RedisHA系统，基本信息如下：

代码位置

https://github.com/double12gzh/redis_HA-keepalived-sentinel-reids

节点个数

master node1个， slave node >= 2个

版本信息

redis 4.0.2，ruby 2.4.2，keepalived1.2.13，tcl 8.6.7，Centos 7.0.1406

功能介绍

结构： redis主从+Sentinel+Keepalived。集群创建完成后，会包含一个master节点和多个slave节点，并且Keepalived会在master节点上绑定一个vip。

当master节点挂掉后，由sentinel进程自动选一个slave并将它变为master，同时Keepalived会把vip挂到新的master上。

         当原来的master节点恢复后，它会自动变为slave节点。

适应范围

本HA Redis高可用系统，可以保证单个redis实例的高可用，适用于业务量比较小的应用。

集群拓扑

 

验证方法

1.    查看各个节点上的redis-server和redis-sentinel进程是否正常启动

[root@redis-ha-redis-master-rj72p3anod5f~]# ps aux | grep redis
root      1797 0.2  0.2 147288  9764 ?       Ssl  20:50   0:00 /usr/local/bin/redis-server0.0.0.0:6379
root      1803 0.2  0.1 145240  7764 ?       Ssl  20:50   0:00 redis-sentinel *:26379 [sentinel]
root      2740 0.0  0.0 112640   976 pts/0   S+   20:54   0:00 grep --color=auto redis
[root@redis-ha-redis-master-rj72p3anod5f~]#

2.    查看节点的角色（master还是slave。此处使用的是默认端口6379，如果使用其它端口并在/etc/redis/redis.conf中加了bind YOUR_BIND_IP，请执行：redis-cli –p YOUR_PORT –h YOUR_BIND_IP）。

Master 节点。从这里我们可以看到它有两个slave节点，分别是slave0和slave1。

[root@redis-ha-redis-master-rj72p3anod5f~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>info replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.16.13.188,port=6379,state=online,offset=64753,lag=1
slave1:ip=172.16.13.187,port=6379,state=online,offset=64753,lag=1
master_replid:1736f8cd46c1eb1f78c4cdef3130f6aa78b3863f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:64753
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:64753
0.0.0.0:6379>

Slave节点。这里可以看到它所属的master节点是172.16.13.186。

[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-0-merb6ycj2llg~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>info replication
# Replication
role:slave
master_host:172.16.13.186
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:87749
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:1736f8cd46c1eb1f78c4cdef3130f6aa78b3863f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:87749
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:87749
0.0.0.0:6379>

查看sentinel是否正常启动redis-cli –p 26379 info sentinel

3.    进行读写测试。

在master节点上执行如下写命令

[root@redis-ha-redis-master-rj72p3anod5f~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>set name "Jeffrey Guan"
OK
0.0.0.0:6379>

在slave节点上执行如下读命令

[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-0-merb6ycj2llg~]# redis-cli -p 6379 -h 0.0.0.0
0.0.0.0:6379>get name
"JeffreyGuan"
0.0.0.0:6379>

4.    测试主从切换

切换前的基本拓扑如图：

 

先看一下master节点上的IP信息，应该可以看到eth0有两个IP，其中172.16.13.185这个是VIP，由keepalive管理。

Master节点：

[root@redis-ha-redis-master-rj72p3anod5f~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
    link/ether fa:16:3e:ea:e0:64 brdff:ff:ff:ff:ff:ff
    inet 172.16.13.186/24 brd 172.16.13.255scope global dynamic eth0
       valid_lft 85813sec preferred_lft85813sec
    inet 172.16.13.185/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feea:e064/64 scopelink
       valid_lft forever preferred_lft forever

Slave节点：

[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-0-merb6ycj2llg~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
    link/ether fa:16:3e:2e:4b:45 brdff:ff:ff:ff:ff:ff
    inet 172.16.13.187/24 brd 172.16.13.255scope global dynamic eth0
       valid_lft 85824sec preferred_lft85824sec
    inet6 fe80::f816:3eff:fe2e:4b45/64 scopelink
       valid_lft forever preferred_lft forever

执行以下命令将master节点上的redis-server关掉，然后查看master节点上的IP信息，发现eth0的172.16.13.185这个IP已不存在了。

[root@redis-ha-redis-master-rj72p3anod5f~]# ps aux | grep redis
root      1797 0.1  0.2 147288  9764 ?       Ssl  20:50  0:01 /usr/local/bin/redis-server 0.0.0.0:6379
root      1803 0.2  0.1 145240  7764 ?       Ssl  20:50   0:01 redis-sentinel *:26379 [sentinel]
root      4554 0.0  0.0 112640   976 pts/0   S+   21:01   0:00 grep --color=auto redis
[root@redis-ha-redis-master-rj72p3anod5f~]# kill -9 1797
[root@redis-ha-redis-master-rj72p3anod5f~]# ps aux  | grep redis
root      1803 0.2  0.1 145240  7764 ?       Ssl  20:50   0:01 redis-sentinel *:26379 [sentinel]
root      4613 0.0  0.0 112640   980 pts/0   S+   21:01   0:00 grep --color=auto redis
[root@redis-ha-redis-master-rj72p3anod5f~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
    link/ether fa:16:3e:ea:e0:64 brd ff:ff:ff:ff:ff:ff
    inet 172.16.13.186/24 brd 172.16.13.255scope global dynamic eth0
       valid_lft 85693sec preferred_lft85693sec
    inet6 fe80::f816:3eff:feea:e064/64 scopelink
       valid_lft forever preferred_lft forever
[root@redis-ha-redis-master-rj72p3anod5f~]#

查看slave节点上的IP信息，发现172.16.13.185这个IP已漂到了此slave节点上，同时发现此slave节点已变成了master的角色。其中，角色的变换是由redis-sentinel控制，vip的漂移由keepalived控制。

[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-1-ftdajerozp4z~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP>mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:68:07:0c brdff:ff:ff:ff:ff:ff
    inet 172.16.13.188/24 brd 172.16.13.255scope global dynamic eth0
       valid_lft 85422sec preferred_lft85422sec
    inet 172.16.13.185/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe68:70c/64 scopelink
       valid_lft forever preferred_lft forever
You have newmail in /var/spool/mail/root
[root@re-s-ha-redis-slave-group-4q2ilw7cdxru-1-ftdajerozp4z~]#

再到原先的master节点上把redis-server启动起来，查看其IP信息及角色，发现它没有了VIP，并且角色变为了slave。

[root@redis-ha-redis-master-rj72p3anod5f~]# redis-cli info replication
# Replication
role:slave
master_host:172.16.13.188
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:210910
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:67b479dee278eddd337e6a40ecfa4965861b32b5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:210910
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:135372
repl_backlog_histlen:75539
[root@redis-ha-redis-master-rj72p3anod5f~]# ip a
1: lo:<LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
    link/ether fa:16:3e:ea:e0:64 brdff:ff:ff:ff:ff:ff
    inet 172.16.13.186/24 brd 172.16.13.255scope global dynamic eth0
       valid_lft 85281sec preferred_lft 85281sec
    inet6 fe80::f816:3eff:feea:e064/64 scopelink
       valid_lft forever preferred_lft forever
[root@redis-ha-redis-master-rj72p3anod5f~]#

切换后的拓扑如图：

关于VIP漂移，可以查看log：/var/log/message

Master节点上，可以看到VIP被移除了。

Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1836]: VRRP_Group(VG_1) Syncinginstances to FAULT state
Sep 30 21:02:26base-server-with-network avahi-daemon[397]: Withdrawing address record for172.16.13.185 on eth0.
Sep 30 21:02:26base-server-with-network Keepalived_healthcheckers[1835]: Netlink reflectorreports IP 172.16.13.185 removed

Slave节点上，可以看到VIP被添加了。

Sep 30 21:02:13base-server-with-network Keepalived_healthcheckers[1837]: Netlink reflectorreports IP 172.16.13.185 removed
Sep 30 21:02:13base-server-with-network avahi-daemon[430]: Withdrawing address record for172.16.13.185 on eth0.
Sep 30 21:02:13base-server-with-network Keepalived_vrrp[1838]: SMTP alert successfully sent.
Sep 30 21:02:25base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) Transitionto MASTER STATE
Sep 30 21:02:25base-server-with-network Keepalived_vrrp[1838]: VRRP_Group(VG_1) Syncinginstances to MASTER state
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) EnteringMASTER STATE
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) settingprotocol VIPs.
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 172.16.13.185
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: Remote SMTP server[127.0.0.1]:25 connected.
Sep 30 21:02:26base-server-with-network Keepalived_healthcheckers[1837]: Netlink reflectorreports IP 172.16.13.185 added
Sep 30 21:02:26base-server-with-network avahi-daemon[430]: Registering new address record for172.16.13.185 on eth0.IPv4.
Sep 30 21:02:26base-server-with-network Keepalived_vrrp[1838]: SMTP alert successfully sent.
Sep 30 21:02:31base-server-with-network Keepalived_vrrp[1838]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 172.16.13.185