使用keepalived打造轻量级的lvs高可用集群

使用keepalived打造轻量级的lvs高可用集群

一、VRRP协议

VRRP: Virtual Routing Redundent Rrotocol  虚拟路由冗余协议

学过网络的朋友都知道，网络在设计的时候必须考虑到冗余容灾，包括线路冗余，设备冗余等，防止网络存在单点故障，那在路由器或三层交换机处实现冗余就显得尤为重要，在网络里面有个协议就是来做这事的，这个协议就是VRRP协议，Keepalived就是巧用VRRP协议来实现高可用性(HA)的。

下图为VRRP协议的工作模型图：

如上图所示，RouterA，B，C组成一个虚拟路由器。此虚拟路由器有自己的IP地址，局域网内的主机将虚拟路由器设置为缺省网关。RouterA、B、C中优先级高的路由器作为Master路由器，承担网关的功能。其余两台路由器作为Backup路由器。

1、备份组中路由器的优先级

VRRP根据优先级来确定备份组中每台路由器的角色(Master路由器或Backup路由器)。优先级越高则越有可能成为Master路由器

VRRP优先级的取值范围为0-255(数值越大优先级越高)，可配置的范围是1-254,0为系统保留，255是系统保留给IP地址拥有者。当路由器为IP地址拥有者时，其优先级始终为255，因此，当备份组内存在IP地址拥有者时，只要其工作正常，则为Master路由器。

2、备份组中的路由器的工作方式

备份组中的路由器具有以下两种工作方式：

非抢占模式：如果备份组中的路由器工作在非抢占模式下，则只要Master路由器没有出现故障，Backup路由器即使随后配置了更高的优先级也不会成为Master路由器。

抢占模式：如果备份组中的路由器工作在抢占模式下，它一旦发现自己的优先级比当前的Master路由器的优先级高，就会对外发送VRRP通告报文，导致备份组内路由器重新选举Master路由器，并最终取代原有的Master路由器。相应的，原来的Master路由器将会变成Backup路由器。

3、备份组中路由器的认证方式

simple：简单字符认证

md5: MD5认证

4、在一个物理设备上，可以配置多个组，靠组ID区别不同的组

具体介绍，参考下面的一篇博客：

http://liangbin332.blog.163.com/blog/static/119684536201051705315390/

二、搭建基于DR模型的lvs集群（web服务集群）

规划图：

1、配置realserver1

[root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0

[root@localhost ~]# service network restart

2、修改部分参数

[root@localhost ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@localhost ~]# echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
[root@localhost ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
[root@localhost ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@localhost ~]# cat /proc/sys/net/ipv4/conf/all/arp_announce
2
[root@localhost ~]# cat /proc/sys/net/ipv4/conf/all/arp_ignore
1

参数介绍：

kernel parameter：

arp_ignore: 接收到ARP请求时的响应级别

默认级别是0：只要本地配置的有相应地址，就予以响应

       1：仅在请求的目标地址配置在请求到达的接口上的时候，才给予响应

arp_announce：定义将自己的地址及MAC地址向外通告时的通告级别

默认级别是0：将本机任何接口上的任何地址向外通告

       1：试图仅向目标网络通告与其网络匹配的地址

       2：仅向与本地接口上地址匹配的网络进行通告

3、在lo:0上配置vip,并配置路由条目

[root@localhost ~]# ifconfig lo:0 172.16.25.1 broadcast 172.16.25.1 netmask 255.255.255.255 up
[root@localhost ~]# route add -host 172.16.25.1 dev lo:0

4、安装web服务，并提供网页

[root@localhost ~]# yum install httpd
[root@localhost ~]# service httpd start
[root@localhost ~]# echo "RS1.lsq.com" > /var/www/html/index.html
[root@localhost html]# curl http://172.16.25.7
RS1.lsq.com

5、ifconfig查看

同样的配置，配置realserver2

三、安装配置keepalived实现lvs的高可用

准备工作：

配置主、从director节点：

1、时间同步（主、从节点时间几个不得超过5秒）

2、主机名要和uname -n 一致，并通过/etc/hosts解析

3、SSH双机互信

配置主节点：
时间同步：
[root@node1 ~]# ntpdate 172.16.0.1      #我们的ftp服务器
主机名要和uname -n 一致，并通过/etc/hosts解析：
[root@node1 ~]# hostname node1.lsq.com
[root@node1 ~]# vim /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.lsq.com
GATEWAY=172.16.0.1
[root@node1 ~]# vim /etc/hosts
添加两行：
172.16.25.10    node1.lsq.com node1
172.16.25.12    node2.lsq.com node2
SSH互信：
配置节点1(主节点)
[root@node1 ~]# ssh-keygen -t rsa -f ~/.ssh/rsa_key -P ''
[root@node1 ~]# ssh-copy-id -i .ssh/rsa_key.pub root@node2.lsq.com  #把公钥发给从节点
1,2两点主从节点配置一致，不在演示
在从节点上建立SSH互信：
[root@node1 ~]# ssh-keygen -t rsa -f ~/.ssh/rsa_key -P ''
[root@node1 ~]# ssh-copy-id -i .ssh/rsa_key.pub root@node1.lsq.com  #把公钥发给从节点

配置过程：

1、为director的主、备节点安装keepalived的rpm包，并提供配置文件

配置主节点

[root@node1 ~]# yum -y --nogpgcheck localinstall keepalived-1.2.7-5.el5.i386.rpm
[root@node1 ~]# scp keepalived-1.2.7-5.el5.i386.rpm node2:/root
[root@node1 ~]# yum install ipvsadm   #安装ipvs规则
[root@node1 keepalived]# cd /etc/keepalived
[root@node1 keepalived]# ls
keepalived.conf  keepalived.conf.haproxy_example  notify.sh
[root@node1 keepalived]# cp keepalived.conf keepalived.conf.bak
[root@node1 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {      #出现故障后向谁发送电子邮件
     root@localhost
   }
   notification_email_from keepalived@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_instance VI_1 {      #定义VRRP实例，即一个虚拟路由组
    state MASTER          #定义在初始状态下谁是主，谁是从
    interface eth0        #在那个接口上配置地址和完成选举，使用ipaddr配置，不具备别名
    virtual_router_id 86  #组ID
    priority 101          #优先级
    advert_int 1          #通告时间间隔
    authentication {      #认证
        auth_type PASS    #PASS表示使用字符串认证
        auth_pass keepalivedpass
    }
    virtual_ipaddress {   #vip
        172.16.25.1/16 dev eth0 label eth0:0
    }
}
virtual_server 172.16.25.1 80 {     #定义集群服务
    delay_loop 6                    #获取某个服务时的等待时长
    lb_algo rr                      #负载均衡调度算法
    lb_kind DR
    nat_mask 255.255.0.0            #负载均衡集群类型
  # persistence_timeout 50          #是不是支持持久链接，这里我们不使用
    protocol TCP                    #tcp协议
    real_server 172.16.25.7 80 {    #realserver
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 1
        }
    }
            connect_timeout 2      #3秒中探测一次健康状况
            nb_get_retry 3         #探测三次都是不健康，就判定不健康
            delay_before_retry 1   #探测一次后，再隔几秒钟后探测
        }
    }
    real_server 172.16.25.8 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 2
            nb_get_retry 3
            delay_before_retry 1
        }
    }
}
[root@node1 keepalived]# scp keepalived.conf node2:/etc/keepalived/
配置node2
[root@node1 ~]# yum -y --nogpgcheck localinstall keepalived-1.2.7-5.el5.i386.rpm
[root@node1 ~]# yum install ipvsadm   #安装ipvs规则
[root@node2 ~]# cd /etc/keepalived/
[root@node2 keepalived]# ls
keepalived.conf  keepalived.conf.haproxy_example  notify.sh
[root@node2 keepalived]# vim keepalived.conf

只需在主节点配置文件的基础上做如下修改即可：

将 state MASTER  改为 state BACKUP

将 priority 101  改为 priority 100

然后在主、备节点上启动keepalived服务

[root@node1 keepalived]# service keepalived start

在director1上执行ifconfig命令

[root@node1 keepalived]# ifconfig

查看ipvs规则

打开浏览器，访问以下vip

刷新页面

2、手动将RS1上的web服务关闭，查看是否能将服务移除

在RS1上关闭web服务
[root@localhost ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
在node1上查看规则
[root@node1 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.25.1:80 rr
  -> 172.16.25.8:80               Route   1      1          0       
You have new mail in /var/spool/mail/root
查看邮件
[root@node1 keepalived]# mail
Mail version 8.1 6/6/93.  Type ? for help.
"/var/spool/mail/root": 6 messages 6 new
>N  1 logwatch@localhost.l  Fri Mar 29 14:53  45/1675  "Logwatch for localhost.localdomain (Linux)"
 N  2 logwatch@localhost.l  Mon May  6 21:46  45/1672  "Logwatch for localhost.localdomain (Linux)"
 N  3 logwatch@localhost.l  Tue May  7 04:02 127/3683  "Logwatch for localhost.localdomain (Linux)"
 N  4 logwatch@localhost.l  Thu May  9 19:35  69/2231  "Logwatch for localhost.localdomain (Linux)"
 N  5 root@localhost.local  Thu May  9 19:41  19/837   "Anacron job for 'localhost.localdomain' cron.daily"
 N  6 keepalived@localhost  Fri May 17 14:32  13/571   "[LVS_DEVEL] Realserver [172.16.25.7]:80 - DOWN"
& 6
Message 6:
From keepalived@localhost.localdomain  Fri May 17 14:32:40 2013
Date: Fri, 17 May 2013 06:32:40 +0000
From: keepalived@localhost.localdomain
Subject: [LVS_DEVEL] Realserver [172.16.25.7]:80 - DOWN
X-Mailer: Keepalived
=> CHECK failed on service : connection error <=

3、当两个realserver都down了，我们怎么办呢？

我们使用director来相应用户请求，在两个director高可用节点上都提供网页页面

director1(主)：
[root@node1 keepalived]# yum install httpd
[root@node1 keepalived]# service httpd start
[root@node1 keepalived]# echo "Realserver is weihuing!" > /var/www/html/index.html
[root@node1 keepalived]# scp /var/www/html/index.html node2:/var/www/html/
在director2上安装web服务，并启动
[root@node1 keepalived]# yum install httpd
[root@node1 keepalived]# service httpd start
在director的主、备节点上，编辑keepalived的配置文件
[root@node1 keepalived]# pwd
/etc/keepalived
[root@node1 keepalived]# vim keepalived.conf
定位至：virtual_server，在{}里面添加一行：
sorry_server 127.0.0.1 80
上述操作主、备节点一样，然后主、备节点都启动keepalived服务，使之生效
[root@node1 keepalived]# service keepalived restart
Stopping keepalived:                                       [  OK  ]
Starting keepalived:                                       [  OK  ]
在realserver1和realserver2上都关闭web服务
[root@localhost ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
然后在director1(主)上查看ipvs规则：
[root@node1 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.25.1:80 rr
  -> 127.0.0.1:80                 Local   1      0          0       
You have new mail in /var/spool/mail/root

从新启动两个realserver上的web服务
[root@localhost ~]# service httpd start
Starting httpd:                                            [  OK  ]
在director1上查看ipvs规则
[root@node1 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.25.1:80 rr
  -> 172.16.25.8:80               Route   1      0          0       
  -> 172.16.25.7:80               Route   1      0          0       
You have new mail in /var/spool/mail/root

3、director主、备节点之间的高可用的实现

[root@node1 keepalived]# vim keepalived.conf
在VRRP实例之外，添加如下内容：
vrrp_script chk_schedown {
    script "[ -e /etc/keepalived/down ] && exit 1 || exit 0"  #如果director主节点这个目录下存在down文件，就down掉主director，否则不down
    interval 1      #检测一次的时间
    weight -5       #如果存在down文件，主节点的优先级减5(减后要小于备节点的优先级才可)
    fall 2          #如果失败了，需要检测2次
    rise 1          #成功只需一次
}
然后还在配置文件中，添加如下内容：
定位至：vrrp_instance VI_1 { ，在vrrp实例中添加如下内容，：
track_script {
        chk_schedown      #定义上述脚本什么时候执行
    }
以上操作director的主备节点的配置文件都要配置，内容一样
而后两个节点上都重启服务
[root@node1 keepalived]# service keepalived restart
Stopping keepalived:                                       [  OK  ]
Starting keepalived:                                       [  OK  ]
在主director的/etc/keepalived目录下，touch一个down文件
[root@node1 keepalived]# pwd
/etc/keepalived
[root@node1 keepalived]# touch down
查看日志信息可以看出vip的漂移
[root@node1 keepalived]# tail /var/log/messages
May 17 19:15:21 localhost Keepalived_healthcheckers[13447]: Netlink reflector reports IP 172.16.25.1 added
May 17 19:15:21 localhost avahi-daemon[3777]: Registering new address record for 172.16.25.1 on eth0.
May 17 19:15:26 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.16.25.1
May 17 19:17:11 localhost Keepalived_vrrp[13448]: VRRP_Script(chk_schedown) failed
May 17 19:17:13 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) Received higher prio advert
May 17 19:17:13 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) Entering BACKUP STATE
May 17 19:17:13 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) removing protocol VIPs.
May 17 19:17:13 localhost Keepalived_vrrp[13448]: Netlink reflector reports IP 172.16.25.1 removed
May 17 19:17:13 localhost Keepalived_healthcheckers[13447]: Netlink reflector reports IP 172.16.25.1 removed
May 17 19:17:13 localhost avahi-daemon[3777]: Withdrawing address record for 172.16.25.1 on eth0.
在director2上执行ifconfig

4、写个脚本实现当vrrp事物发生时(主备节点切换)，发送警报邮件给指定的管理员

在/etc/keepalived下编写一个脚本
[root@node1 keepalived]# pwd
/etc/keepalived
[root@node1 keepalived]# vim new_notify.sh

#!/bin/bash
#
contact='root@localhost'
Usage() {
        echo "Usage: `basename $0` {master|backup|fault} VIP"
}
Notify() {
        subject="`hostname`'s state changed to $1"
        mailbody="`date "+%F %T"`: `hostname`'s state change to $1, $VIP floating."
        echo $mailbody | mail -s "$subject" $contact
}
[ $# -lt 2 ] && Usage && exit
VIP=$2
case $1 in
  master)
        Notify master
        ;;
  backup)
        Notify backup
        ;;
  fault)
        Notify fault
        ;;
  *)
        Usage
        exit 1
        ;;
esac

[root@node1 keepalived]# chmod +x new_notify.sh
[root@node1 keepalived]# scp -p new_notify.sh node2:/etc/keepalived/    #复制到另外的节点上一份

配置director的两个节点上keepalived的配置文件：

[root@node1 keepalived]# vim keepalived.conf

定位至：vrrp_instance VI_1 { ，在vrrp实例中添加如下内容，：
notify_master "/etc/keepalived/new_notify.sh master 172.16.25.1"
notify_backup "/etc/keepalived/new_notify.sh backup 172.16.25.1"
notify_fault "/etc/keepalived/new_notify.sh fault 172.16.25.1"
两个节点都启动keepalived服务
[root@node1 keepalived]# service keepalived restart
Stopping keepalived:                                       [  OK  ]
Starting keepalived:                                       [  OK  ]
在director1上删除down文件
[root@node1 keepalived]# ls
down  keepalived.conf  keepalived.conf.bak  keepalived.conf.haproxy_example  new_notify.sh  notify.sh
[root@node1 keepalived]# rm -rf down
[root@node1 keepalived]# tail /var/log/maillog      #可以根据邮件日志，查询邮件发送状态

执行mail，查看邮件

在director1上ifconfig

现在我们已经使用keepalived实现了lvs的高可用了，并且我们还自写脚本实现director主、备节点切换时可以发送警报邮件信息，是不是很easy，大家都来试试吧！

转载于:https://blog.51cto.com/crazylsq/1202972