理论部分
高可用集群KEEPALIVED是一种用于实现集群高可用性的服务软件,主要用于防止单点故障,确保集群中的服务能够持续、稳定地对外提供服务。以下是对KEEPALIVED的详细解析:
一、基本概念
-
定义:KEEPALIVED基于VRRP(Virtual Router Redundancy Protocol,虚拟路由冗余协议)实现,是集群管理中保证集群高可用性的关键组件。
-
作用:KEEPALIVED通过检测集群中各节点的状态,并在必要时进行故障转移,以确保集群服务的连续性。
二、工作原理
-
VRRP协议:KEEPALIVED以VRRP协议为实现基础,该协议允许多台路由器组成一个虚拟路由器组,通过选举产生一个主路由器(Master)和多个备份路由器(Backup)。主路由器负责对外提供服务,并周期性地发送VRRP报文以维持其状态。
-
故障检测与转移:当BACKUP主机收不到MASTER主机的VRRP报文时,会认为MASTER主机宕机,此时会根据VRRP的优先级选举一个新的MASTER主机接替服务。
三、主要模块
-
core模块:KEEPALIVED的核心模块,负责主进程的启动、维护以及全局配置文件的加载和解析。
-
check模块:负责健康检查,包括各种常见的检查方式,如ICMP、TCP端口状态、HTTP GET等,以确保集群中各节点的正常运行。
-
vrrp模块:实现VRRP协议的关键模块,负责虚拟路由器的选举、状态维护和报文发送等工作。
四、应用场景
-
LVS高可用架构:KEEPALIVED常用于LVS(Linux Virtual Server,Linux虚拟服务器)集群中,作为调度器的高可用解决方案。通过设置主调度器和备调度器,确保在主调度器故障时能够迅速切换至备调度器,从而保证服务的连续性。
-
其他高可用场景:除了LVS集群外,KEEPALIVED还可用于其他需要高可用性的场景,如数据库集群、Web服务器集群等。
五、注意事项
-
脑裂问题:在高可用架构中,可能会出现脑裂问题,即主和备节点都认为自己是主节点,并同时提供服务。这通常是由于网络问题、防火墙策略等原因导致的。为了避免脑裂问题,可以采取多种措施,如添加更多的检测手段、使用串行电缆和以太网电缆连接等。
-
配置与监控:在使用KEEPALIVED时,需要合理配置其参数和监控其状态,以确保其能够正常工作并及时发现潜在问题。
实验部分
一、Keepalived基本部署
1、基本配置
#环境准备 四台都用rh7 kA1: 172.25.254.10 KA2: 172.25.254.20 realserver1: 172.25.254.110 realserver2: 172.25.254.120 #realserver1、realserver2 yum install httpd -y [root@realserver1 ~]# echo realserver1 - 172.25.254.110 > /var/www/html/index.html [root@realserver1 ~]# systemctl enable --now httpd [root@realserver2 ~]# echo realserver2 - 172.25.254.120 > /var/www/html/index.html [root@realserver2 ~]# systemctl enable --now httpd #ka1、ka2访问 curl 172.25.254.110 curl 172.25.254.120
2、虚拟路由管理
#ka1、ka2 [root@ka1 ~]# yum install keepalived -y #全局配置 [root@ka1 ~]# vim /etc/keepalived/keepalived.conf global_defs { notification_email { 2727584440@qq.com } notification_email_from keepalived@timinglee.org smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka1.timinglee.org vrrp_skip_check_adv_addr vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 vrrp_mcast_group4 224.0.0.18 } #虚拟路由器配置 vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } } [root@ka1 ~]# systemctl enable --now keepalived.service [root@ka1 ~]# ifconfig [root@ka1 ~]# scp /etc/keepalived/keepalived.conf root@172.25.254.20:/etc/keepalived/keepalived.conf # 在ka2上修改这两条,先不重新启动,查看是那个被用 vrrp_instance VI_1 { state BACKUP priority 80 #测试 [root@ka1 ~]# tcpdump -i eth0 -nn host 224.0.0.18 23:04:18.883958 IP 172.25.254.10 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20 #关掉ka1的keepalived,会出现另一台的 23:04:05.872730 IP 172.25.254.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
3、ping172.25.254.10能通
同时关闭则不能ping通,同时开启则可以
# ka1、2 [root@ka1 ~]# vim /etc/keepalived/keepalived.conf global_defs { notification_email { 2727584440@qq.com } notification_email_from keepalived@timinglee.org smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka1.timinglee.org vrrp_skip_check_adv_addr vrrp_strict #和vrrp_iptables 同时开启同时关闭 vrrp_garp_interval 0 vrrp_gna_interval 0 vrrp_mcast_group4 224.0.0.18 vrrp_iptables # }
4、启用keepalived日志功能
[root@ka1 ~]# vim /etc/sysconfig/keepalived KEEPALIVED_OPTIONS="-D -S 6" [root@ka1 ~]# vim /etc/rsyslog.conf local6.* /var/log/keepa lived.log [root@ka1 ~]# systemctl restart keepalived.service [root@ka1 ~]# systemctl restart rsyslog.service [root@ka1 ~]# ll /var/log/keepalived.log
5、独立子配置文件
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf #vrrp_instance VI_1 { # state MASTER # interface eth0 # virtual_router_id 100 # priority 100 # advert_int 1 # authentication { # auth_type PASS # auth_pass 1111 # } # virtual_ipaddress { # 172.25.254.100/24 dev eth0 label eth0:1 # } #} include "/etc/keepalived/conf.d/*.conf" [root@ka1 ~]# mkdir -p /etc/keepalived/conf.d [root@ka1 ~]# vim /etc/keepalived/conf.d/172.25.254.100.conf vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } } [root@ka1 ~]# systemctl restart keepalived.service [root@ka1 ~]# ifconfig
二、抢占模式和非抢占模式
默认为抢占模式preempt,即当高优先级的主机恢复在线后,会抢占低先级的主机的master角色,
这样会使vip在KA主机中来回漂移,造成网络抖动,
建议设置为非抢占模式 nopreempt ,即高优先级主机恢复后,并不会抢占低优先级主机的master角色
非抢占模块下,如果原主机down机, VIP迁移至的新主机, 后续也发生down时,仍会将VIP迁移回原主机
1、非抢占模式
#注意:要关闭 VIP抢占,必须将各 keepalived 服务器state配置为BACKUP #ka1、ka2主机配置 vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 100 #优先级高 advert_int 1 nopreempt #非抢占模式 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } } # 测试 关掉ka1,100在ka2,开启ka1,100还在ka2上
2、抢占延迟模式preempt_delay
抢占延迟模式,即优先级高的主机恢复后,不会立即抢回VIP,而是延迟一段时间(默认300s)再抢回VIP
# 注意:需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict #ka1、ka2主机配置 vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 100 advert_int 1 preempt_delay 5s #抢占延迟5S authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } }
三、VIP的单播配置
默认keepalived主机之间利用多播相互通告消息,会造成网络拥塞,可以替换成单播,减少网络流量
# 注意:启用 vrrp_strict 时,不能启用单播 #ka1主机配置 global_defs { notification_email { 2727584440@qq.com } notification_email_from keepalived@timinglee.org smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id ka1.timinglee.org vrrp_skip_check_adv_addr #vrrp_strict #注释此参数,与vip单播模式冲突 vrrp_garp_interval 0 vrrp_gna_interval 0 vrrp_mcast_group4 224.0.0.18 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 100 advert_int 1 preempt_delay 5s authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 #本机IP unicast_peer { 172.25.254.20 #对端IP } } #ka2主机配置 unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } #抓包查看单播效果 [root@ka1 ~]# tcpdump -i eth0 -nn src host 172.25.254.10 and dst 172.25.254.20 [root@ka2 ~]# tcpdump -i eth0 -nn src host 172.25.254.20 and dst 172.25.254.10 # 只有VIP在哪,哪才能去发,如果把ka1停掉,那么ka2才能发,如果把ka1又开启,那么ka2不能发单播,只有ka1能发
四、Keepalived 通知脚本配置
1、 邮件配置
# 安装邮件发送工具 [root@KA1 ~]# yum install mailx -y [root@KA1 ~]# vim /etc/mail.rc #######mail set########## set from=2727584440@qq.com set smtp=smtp.qq.com set smtp-auth-user=2727584440@qq.com set smtp-auth-password=oyulniiqkxridgag #自己邮箱的授权码 set smtp-auth=login set ssl-verify=ignore #发送邮件 echo hello word | mail -s test 2727584440@qq.com
2、实现Keepalived状态切换的通知脚本
#在所有 keepalived节点配置如下 [root@KA1 + KA2 ~]# vim /etc/keepalived/mail.sh #!/bin/bash mail_dst="2727584440@qq.com" send_message() { mail_sub="$HOSTNMAE to be $1 vip move" mail_msg="`date +%F\ %T`: vrrp move $HOSTNAME change $1" echo $mail_msg | mail -s "$mail_sub" $mail_dst } case $1 in master) send_message master ;; backup) send_message backup ;; fault) send_message fault ;; *) ;; esac [root@KA1 +KA2 ~]# chmod +x /etc/keepalived/mail.sh [root@KA1 +K2 ~]#vim /etc/keepalived/keepalived.conf vrrp_instance VI_1 { ... unicast_src_ip 172.25.254.10 #本机IP unicast_peer { 172.25.254.20 #对端IP } notify_master "/etc/keepalived/mail.sh master" notify_backup "/etc/keepalived/mail.sh backup" notify_fault "/etc/keepalived/mail.sh fault" }
五、实现 master/master 的 Keepalived 双主架构
# ka1备 vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } track_script { check_haproxy } } vrrp_instance VI_2 { state BACKUP interface eth0 virtual_router_id 200 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev eth0 label eth0:2 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } } # ka2 vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 100 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } } vrrp_instance VI_2 { state MASTER interface eth0 virtual_router_id 200 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.200/24 dev eth0 label eth0:2 } unicast_src_ip 172.25.254.20 unicast_peer { 172.25.254.10 } }
六、实现IPVS的高可用性
# 在realserver1\2 [root@realserver2 ~]# ip a a 172.25.254.100/32 dev lo [root@realserver2 ~]# vim /etc/sysctl.d/arp.conf net.ipv4.conf.all.arp_ignore=1 net.ipv4.conf.all.arp_announce=2 net.ipv4.conf.lo.arp_ignore=1 net.ipv4.conf.lo.arp_announce=2 [root@realserver2 ~]# sysctl --system # ka1\2 [root@ka1 ~]# yum install ipvsadm -y [root@ka1 ~]# vim /etc/keepalived/keepalived.conf virtual_server 172.25.254.100 80 { delay_loop 6 lb_algo wrr lb_kind DR #persistence_timeout 50 protocol TCP real_server 172.25.254.110 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 2 delay_before_retry 2 } } real_server 172.25.254.120 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 2 delay_before_retry 2 } } } [root@ka1 ~]# systemctl restart keepalived.service [root@ka1 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn [root@ka2 ~]# ipvsadm -A -t 172.255.254.100:80 -s wrr [root@ka2 ~]# ipvsadm -Ln (有) [root@ka1 ~]# ipvsadm -Ln (有) # 测试 [root@ka2 ~]# for i in {1..6}; do curl 172.25.254.100; done realserver2 - 172.25.254.120 realserver1 - 172.25.254.110 realserver2 - 172.25.254.120 realserver1 - 172.25.254.110 realserver2 - 172.25.254.120 realserver1 - 172.25.254.110
七、实现其它应用的高可用性VRRP Script
keepalived利用 VRRP Script 技术,可以调用外部的辅助脚本进行资源监控,并根据监控的结果实现优先
动态调整,从而实现其它应用的高可用性功能
分两步实现:
1、定义脚本
vrrp_script:自定义资源监控脚本,vrrp实例根据脚本返回值,公共定义,可被多个实例调用,定义在vrrp实例之外的独立配置块,一般放在global_defs设置块之后。
通常此脚本用于监控指定应用的状态。一旦发现应用的状态异常,则触发对MASTER节点的权重减至低于SLAVE节点,从而实现 VIP 切换到 SLAVE 节点
vrrp_script <SCRIPT_NAME> { script <STRING>|<QUOTED-STRING> #此脚本返回值为非0时,会触发下面OPTIONS执行 OPTIONS }
2、调用脚本
track_script:调用vrrp_script定义的脚本去监控资源,定义在VRRP实例之内,调用事先定义的
vrrp_script
track_script { SCRIPT_NAME_1 SCRIPT_NAME_2 }
3、实战案例:利用脚本实现主从角色切换
# 在ka1上 [root@ka1 ~]# vim /etc/keepalived/test.sh #!/bin/bash [ ! -f /mnt/lee ] [root@ka1 ~]# sh /etc/keepalived/test.sh [root@ka1 ~]# echo $? 0 [root@ka1 ~]# chmod +x /etc/keepalived/test.sh [root@ka1 ~]# vim /etc/keepalived/keepalived.conf vrrp_script check_haproxy { script "/etc/keepalived/test.sh" interval 1 weight -30 fall 2 rise 2 timeout 2 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } track_script { check_haproxy } } [root@ka1 ~]# systemctl restart keepalived.service # 测试 [root@ka1 ~]# touch /mnt/lee [root@ka1 ~]# ifconfig [root@ka2 ~]# ifconfig eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0 ether 00:0c:29:44:a9:9f txqueuelen 1000 (Ethernet) [root@ka1 ~]# rm -rf /mnt/lee [root@ka1 ~]# systemctl restart keepalived.service [root@ka1 ~]# ifconfig eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0 ether 00:0c:29:42:39:a7 txqueuelen 1000 (Ethernet)
4、实战案例:实现HAProxy高可用
#ka1、ka2 [root@ka1 ~]# yum install haproxy -y [root@ka1 ~]# vim /etc/sysctl.conf net.ipv4.ip_nonlocal_bind = 1 [root@ka1 ~]# sysctl -p [root@ka1 ~]# vim /etc/haproxy/haproxy.cfg listen webcluster bind 172.25.254.100:80 mode http balance roundrobin server web1 172.25.254.110:80 check inter 3 fall 2 rise 5 server web2 172.25.254.120:80 check inter 3 fall 2 rise 5 [root@ka1 ~]# sysetemctl enable ---now haproxy [root@ka1 ~]# netstat -antlupe | grep haproxy [root@ka1 ~]# curl 172.25.254.100 (不能,和lvs冲突) [root@ka1 ~]# vim /etc/keepalived/keepalived.conf 注释掉 #virtual_server 172.25.254.100 80 { [root@ka1 ~]# systemctl restart keepalived.service #realwebser1、2 [root@realserver2 ~]# systemctl restart network (删掉lo) [root@realserver2 ~]# vim /etc/sysctl.d/arp.conf net.ipv4.conf.all.arp_ignore=0 net.ipv4.conf.all.arp_announce=0 net.ipv4.conf.lo.arp_ignore=0 net.ipv4.conf.lo.arp_announce=0 [root@realserver2 ~]# sysctl --system [root@ka1 ~]# curl 172.25.254.100 [root@ka1 ~]# curl 172.25.254.110 [root@ka1 ~]# curl 172.25.254.120 #现在 在ka1关掉haproxy,则在外面访问不了172.25.254.100,我们需要解决这个问题
# 在ka1 [root@ka1 ~]# vim /etc/keepalived/test.sh #!/bin/bash killall -0 haproxy [root@ka1 ~]# sh /etc/keepalived/test.sh [root@ka1 ~]# echo $? [root@ka1 ~]# vim /etc/keepalived/keepalived.conf vrrp_script check_haproxy { script "/etc/keepalived/test.sh" interval 1 weight -30 fall 2 rise 2 timeout 2 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 100 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 172.25.254.100/24 dev eth0 label eth0:1 } unicast_src_ip 172.25.254.10 unicast_peer { 172.25.254.20 } track_script { check_haproxy } } [root@ka1 ~]# systemctl restart keepalived.service # 测试 关掉ka1的haproxy,也能一直访问 [root@ka1 ~]# systemctl stop haproxy.service [root@ka1 ~]# systemctl start haproxy.service [root@ka2 ~]# while true; do curl 172.25.254.100; done