使用keepalived打造轻量级的lvs高可用集群
一、VRRP协议
VRRP: Virtual Routing Redundent Rrotocol 虚拟路由冗余协议
学过网络的朋友都知道,网络在设计的时候必须考虑到冗余容灾,包括线路冗余,设备冗余等,防止网络存在单点故障,那在路由器或三层交换机处实现冗余就显得尤为重要,在网络里面有个协议就是来做这事的,这个协议就是VRRP协议,Keepalived就是巧用VRRP协议来实现高可用性(HA)的。
下图为VRRP协议的工作模型图:
如上图所示,RouterA,B,C组成一个虚拟路由器。此虚拟路由器有自己的IP地址,局域网内的主机将虚拟路由器设置为缺省网关。RouterA、B、C中优先级高的路由器作为Master路由器,承担网关的功能。其余两台路由器作为Backup路由器。
1、备份组中路由器的优先级
VRRP根据优先级来确定备份组中每台路由器的角色(Master路由器或Backup路由器)。优先级越高则越有可能成为Master路由器
VRRP优先级的取值范围为0-255(数值越大优先级越高),可配置的范围是1-254,0为系统保留,255是系统保留给IP地址拥有者。当路由器为IP地址拥有者时,其优先级始终为255,因此,当备份组内存在IP地址拥有者时,只要其工作正常,则为Master路由器。
2、备份组中的路由器的工作方式
备份组中的路由器具有以下两种工作方式:
非抢占模式:如果备份组中的路由器工作在非抢占模式下,则只要Master路由器没有出现故障,Backup路由器即使随后配置了更高的优先级也不会成为Master路由器。
抢占模式:如果备份组中的路由器工作在抢占模式下,它一旦发现自己的优先级比当前的Master路由器的优先级高,就会对外发送VRRP通告报文,导致备份组内路由器重新选举Master路由器,并最终取代原有的Master路由器。相应的,原来的Master路由器将会变成Backup路由器。
3、备份组中路由器的认证方式
simple:简单字符认证
md5: MD5认证
4、在一个物理设备上,可以配置多个组,靠组ID区别不同的组
具体介绍,参考下面的一篇博客:
http://liangbin332.blog.163.com/blog/static/119684536201051705315390/
二、搭建基于DR模型的lvs集群(web服务集群)
规划图:
1、配置realserver1
[root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
[root@localhost ~]# service network restart
2、修改部分参数
[root@localhost ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce [root@localhost ~]# echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce [root@localhost ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore [root@localhost ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore [root@localhost ~]# cat /proc/sys/net/ipv4/conf/all/arp_announce 2 [root@localhost ~]# cat /proc/sys/net/ipv4/conf/all/arp_ignore 1
参数介绍:
kernel parameter:
arp_ignore: 接收到ARP请求时的响应级别
默认级别是0:只要本地配置的有相应地址,就予以响应
1:仅在请求的目标地址配置在请求到达的接口上的时候,才给予响应
arp_announce:定义将自己的地址及MAC地址向外通告时的通告级别
默认级别是0:将本机任何接口上的任何地址向外通告
1:试图仅向目标网络通告与其网络匹配的地址
2:仅向与本地接口上地址匹配的网络进行通告
3、在lo:0上配置vip,并配置路由条目
[root@localhost ~]# ifconfig lo:0 172.16.25.1 broadcast 172.16.25.1 netmask 255.255.255.255 up [root@localhost ~]# route add -host 172.16.25.1 dev lo:0
4、安装web服务,并提供网页
[root@localhost ~]# yum install httpd [root@localhost ~]# service httpd start [root@localhost ~]# echo "RS1.lsq.com" > /var/www/html/index.html [root@localhost html]# curl http://172.16.25.7 RS1.lsq.com
5、ifconfig查看
同样的配置,配置realserver2
三、安装配置keepalived实现lvs的高可用
准备工作:
配置主、从director节点:
1、时间同步(主、从节点时间几个不得超过5秒)
2、主机名要和uname -n 一致,并通过/etc/hosts解析
3、SSH双机互信
配置主节点: 时间同步: [root@node1 ~]# ntpdate 172.16.0.1 #我们的ftp服务器 主机名要和uname -n 一致,并通过/etc/hosts解析: [root@node1 ~]# hostname node1.lsq.com [root@node1 ~]# vim /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=node1.lsq.com GATEWAY=172.16.0.1 [root@node1 ~]# vim /etc/hosts 添加两行: 172.16.25.10 node1.lsq.com node1 172.16.25.12 node2.lsq.com node2 SSH互信: 配置节点1(主节点) [root@node1 ~]# ssh-keygen -t rsa -f ~/.ssh/rsa_key -P '' [root@node1 ~]# ssh-copy-id -i .ssh/rsa_key.pub root@node2.lsq.com #把公钥发给从节点 1,2两点主从节点配置一致,不在演示 在从节点上建立SSH互信: [root@node1 ~]# ssh-keygen -t rsa -f ~/.ssh/rsa_key -P '' [root@node1 ~]# ssh-copy-id -i .ssh/rsa_key.pub root@node1.lsq.com #把公钥发给从节点
配置过程:
1、为director的主、备节点安装keepalived的rpm包,并提供配置文件
配置主节点
[root@node1 ~]# yum -y --nogpgcheck localinstall keepalived-1.2.7-5.el5.i386.rpm [root@node1 ~]# scp keepalived-1.2.7-5.el5.i386.rpm node2:/root [root@node1 ~]# yum install ipvsadm #安装ipvs规则 [root@node1 keepalived]# cd /etc/keepalived [root@node1 keepalived]# ls keepalived.conf keepalived.conf.haproxy_example notify.sh [root@node1 keepalived]# cp keepalived.conf keepalived.conf.bak [root@node1 keepalived]# vim keepalived.conf ! Configuration File for keepalived global_defs { notification_email { #出现故障后向谁发送电子邮件 root@localhost } notification_email_from keepalived@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { #定义VRRP实例,即一个虚拟路由组 state MASTER #定义在初始状态下谁是主,谁是从 interface eth0 #在那个接口上配置地址和完成选举,使用ipaddr配置,不具备别名 virtual_router_id 86 #组ID priority 101 #优先级 advert_int 1 #通告时间间隔 authentication { #认证 auth_type PASS #PASS表示使用字符串认证 auth_pass keepalivedpass } virtual_ipaddress { #vip 172.16.25.1/16 dev eth0 label eth0:0 } } virtual_server 172.16.25.1 80 { #定义集群服务 delay_loop 6 #获取某个服务时的等待时长 lb_algo rr #负载均衡调度算法 lb_kind DR nat_mask 255.255.0.0 #负载均衡集群类型 # persistence_timeout 50 #是不是支持持久链接,这里我们不使用 protocol TCP #tcp协议 real_server 172.16.25.7 80 { #realserver weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 2 nb_get_retry 3 delay_before_retry 1 } } connect_timeout 2 #3秒中探测一次健康状况 nb_get_retry 3 #探测三次都是不健康,就判定不健康 delay_before_retry 1 #探测一次后,再隔几秒钟后探测 } } real_server 172.16.25.8 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 2 nb_get_retry 3 delay_before_retry 1 } } } [root@node1 keepalived]# scp keepalived.conf node2:/etc/keepalived/ 配置node2 [root@node1 ~]# yum -y --nogpgcheck localinstall keepalived-1.2.7-5.el5.i386.rpm [root@node1 ~]# yum install ipvsadm #安装ipvs规则 [root@node2 ~]# cd /etc/keepalived/ [root@node2 keepalived]# ls keepalived.conf keepalived.conf.haproxy_example notify.sh [root@node2 keepalived]# vim keepalived.conf
只需在主节点配置文件的基础上做如下修改即可:
将 state MASTER 改为 state BACKUP
将 priority 101 改为 priority 100
然后在主、备节点上启动keepalived服务
[root@node1 keepalived]# service keepalived start
在director1上执行ifconfig命令
[root@node1 keepalived]# ifconfig
查看ipvs规则
打开浏览器,访问以下vip
刷新页面
2、手动将RS1上的web服务关闭,查看是否能将服务移除
在RS1上关闭web服务 [root@localhost ~]# service httpd stop Stopping httpd: [ OK ] 在node1上查看规则 [root@node1 keepalived]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.25.1:80 rr -> 172.16.25.8:80 Route 1 1 0 You have new mail in /var/spool/mail/root 查看邮件 [root@node1 keepalived]# mail Mail version 8.1 6/6/93. Type ? for help. "/var/spool/mail/root": 6 messages 6 new >N 1 logwatch@localhost.l Fri Mar 29 14:53 45/1675 "Logwatch for localhost.localdomain (Linux)" N 2 logwatch@localhost.l Mon May 6 21:46 45/1672 "Logwatch for localhost.localdomain (Linux)" N 3 logwatch@localhost.l Tue May 7 04:02 127/3683 "Logwatch for localhost.localdomain (Linux)" N 4 logwatch@localhost.l Thu May 9 19:35 69/2231 "Logwatch for localhost.localdomain (Linux)" N 5 root@localhost.local Thu May 9 19:41 19/837 "Anacron job for 'localhost.localdomain' cron.daily" N 6 keepalived@localhost Fri May 17 14:32 13/571 "[LVS_DEVEL] Realserver [172.16.25.7]:80 - DOWN" & 6 Message 6: From keepalived@localhost.localdomain Fri May 17 14:32:40 2013 Date: Fri, 17 May 2013 06:32:40 +0000 From: keepalived@localhost.localdomain Subject: [LVS_DEVEL] Realserver [172.16.25.7]:80 - DOWN X-Mailer: Keepalived => CHECK failed on service : connection error <=
3、当两个realserver都down了,我们怎么办呢?
我们使用director来相应用户请求,在两个director高可用节点上都提供网页页面
director1(主): [root@node1 keepalived]# yum install httpd [root@node1 keepalived]# service httpd start [root@node1 keepalived]# echo "Realserver is weihuing!" > /var/www/html/index.html [root@node1 keepalived]# scp /var/www/html/index.html node2:/var/www/html/ 在director2上安装web服务,并启动 [root@node1 keepalived]# yum install httpd [root@node1 keepalived]# service httpd start 在director的主、备节点上,编辑keepalived的配置文件 [root@node1 keepalived]# pwd /etc/keepalived [root@node1 keepalived]# vim keepalived.conf 定位至:virtual_server,在{}里面添加一行: sorry_server 127.0.0.1 80 上述操作主、备节点一样,然后主、备节点都启动keepalived服务,使之生效 [root@node1 keepalived]# service keepalived restart Stopping keepalived: [ OK ] Starting keepalived: [ OK ] 在realserver1和realserver2上都关闭web服务 [root@localhost ~]# service httpd stop Stopping httpd: [ OK ] 然后在director1(主)上查看ipvs规则: [root@node1 keepalived]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.25.1:80 rr -> 127.0.0.1:80 Local 1 0 0 You have new mail in /var/spool/mail/root
从新启动两个realserver上的web服务 [root@localhost ~]# service httpd start Starting httpd: [ OK ] 在director1上查看ipvs规则 [root@node1 keepalived]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.25.1:80 rr -> 172.16.25.8:80 Route 1 0 0 -> 172.16.25.7:80 Route 1 0 0 You have new mail in /var/spool/mail/root
3、director主、备节点之间的高可用的实现
[root@node1 keepalived]# vim keepalived.conf 在VRRP实例之外,添加如下内容: vrrp_script chk_schedown { script "[ -e /etc/keepalived/down ] && exit 1 || exit 0" #如果director主节点这个目录下存在down文件,就down掉主director,否则不down interval 1 #检测一次的时间 weight -5 #如果存在down文件,主节点的优先级减5(减后要小于备节点的优先级才可) fall 2 #如果失败了,需要检测2次 rise 1 #成功只需一次 } 然后还在配置文件中,添加如下内容: 定位至:vrrp_instance VI_1 { ,在vrrp实例中添加如下内容,: track_script { chk_schedown #定义上述脚本什么时候执行 } 以上操作director的主备节点的配置文件都要配置,内容一样 而后两个节点上都重启服务 [root@node1 keepalived]# service keepalived restart Stopping keepalived: [ OK ] Starting keepalived: [ OK ] 在主director的/etc/keepalived目录下,touch一个down文件 [root@node1 keepalived]# pwd /etc/keepalived [root@node1 keepalived]# touch down 查看日志信息可以看出vip的漂移 [root@node1 keepalived]# tail /var/log/messages May 17 19:15:21 localhost Keepalived_healthcheckers[13447]: Netlink reflector reports IP 172.16.25.1 added May 17 19:15:21 localhost avahi-daemon[3777]: Registering new address record for 172.16.25.1 on eth0. May 17 19:15:26 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.16.25.1 May 17 19:17:11 localhost Keepalived_vrrp[13448]: VRRP_Script(chk_schedown) failed May 17 19:17:13 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) Received higher prio advert May 17 19:17:13 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) Entering BACKUP STATE May 17 19:17:13 localhost Keepalived_vrrp[13448]: VRRP_Instance(VI_1) removing protocol VIPs. May 17 19:17:13 localhost Keepalived_vrrp[13448]: Netlink reflector reports IP 172.16.25.1 removed May 17 19:17:13 localhost Keepalived_healthcheckers[13447]: Netlink reflector reports IP 172.16.25.1 removed May 17 19:17:13 localhost avahi-daemon[3777]: Withdrawing address record for 172.16.25.1 on eth0. 在director2上执行ifconfig
4、写个脚本实现当vrrp事物发生时(主备节点切换),发送警报邮件给指定的管理员
在/etc/keepalived下编写一个脚本 [root@node1 keepalived]# pwd /etc/keepalived [root@node1 keepalived]# vim new_notify.sh
#!/bin/bash # contact='root@localhost' Usage() { echo "Usage: `basename $0` {master|backup|fault} VIP" } Notify() { subject="`hostname`'s state changed to $1" mailbody="`date "+%F %T"`: `hostname`'s state change to $1, $VIP floating." echo $mailbody | mail -s "$subject" $contact } [ $# -lt 2 ] && Usage && exit VIP=$2 case $1 in master) Notify master ;; backup) Notify backup ;; fault) Notify fault ;; *) Usage exit 1 ;; esac
[root@node1 keepalived]# chmod +x new_notify.sh [root@node1 keepalived]# scp -p new_notify.sh node2:/etc/keepalived/ #复制到另外的节点上一份
配置director的两个节点上keepalived的配置文件:
[root@node1 keepalived]# vim keepalived.conf
定位至:vrrp_instance VI_1 { ,在vrrp实例中添加如下内容,: notify_master "/etc/keepalived/new_notify.sh master 172.16.25.1" notify_backup "/etc/keepalived/new_notify.sh backup 172.16.25.1" notify_fault "/etc/keepalived/new_notify.sh fault 172.16.25.1" 两个节点都启动keepalived服务 [root@node1 keepalived]# service keepalived restart Stopping keepalived: [ OK ] Starting keepalived: [ OK ] 在director1上删除down文件 [root@node1 keepalived]# ls down keepalived.conf keepalived.conf.bak keepalived.conf.haproxy_example new_notify.sh notify.sh [root@node1 keepalived]# rm -rf down [root@node1 keepalived]# tail /var/log/maillog #可以根据邮件日志,查询邮件发送状态
执行mail,查看邮件
在director1上ifconfig
现在我们已经使用keepalived实现了lvs的高可用了,并且我们还自写脚本实现director主、备节点切换时可以发送警报邮件信息,是不是很easy,大家都来试试吧!
转载于:https://blog.51cto.com/crazylsq/1202972