高可用集群KEEPALIVED

一.高可用集群

1.1 集群类型

  • LB:Load Balance 负载均衡

    • LVS/HAProxy/nginx(http/upstream, stream/upstream)
  • HA:High Availability 高可用集群

    • 数据库、Redis
  • SPoF: Single Point of Failure,解决单点故障

    • HPC:High Performance Computing 高性能集群

1.2 系统可用性

SLA:Service-Level Agreement 服务等级协议(提供服务的企业与客户之间就服务的品质、水准、性能

等方面所达成的双方共同认可的协议或契约)

A = MTBF / (MTBF+MTTR)

指标 :99.9%, 99.99%, 99.999%,99.9999%

1.3 系统故障

硬件故障:设计缺陷、wear out(损耗)、非人为不可抗拒因素

软件故障:设计缺陷 bug

1.4 实现高可用

提升系统高用性的解决方案:降低MTTR- Mean Time To Repair(平均故障时间)

解决方案:建立冗余机制

  • active/passive 主/备
  • active/active 双主
  • active --> HEARTBEAT --> passive
  • active <–> HEARTBEAT <–> active

1.5.VRRP:Virtual Router Redundancy Protocol

虚拟路由冗余协议,解决静态网关单点风险

  • 物理层:路由器、三层交换机
  • 软件层:keepalived

1.5.1 VRRP 相关术语

  • 虚拟路由器:Virtual Router
  • 虚拟路由器标识:VRID(0-255),唯一标识虚拟路由器
  • VIP:Virtual IP
  • VMAC:Virutal MAC (00-00-5e-00-01-VRID)
  • 物理路由器:
    • master:主设备
    • backup:备用设备
    • priority:优先级

1.5.2 VRRP 相关技术

通告:心跳,优先级等;周期性

工作方式:抢占式,非抢占式

安全认证:

  • 无认证
  • 简单字符认证:预共享密钥
  • MD5

工作模式:

  • 主/备:单虚拟路由器
  • 主/主:主/备(虚拟路由器1),备/主(虚拟路由器2)

二.Keepalived基础

实现master/slave的keepalived单主架构实验

实验准备:准备四台rhel7.9虚拟机

根据拓扑图配置ip,并关闭防火墙和selinux

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

1.安装keepalived,http

[root@ka1 ~] yum install keepalived.x86_64 -y
[root@ka2 ~] yum install keepalived.x86_64 -y
[root@rs1 ~] yum install httpd -y
[root@rs2 ~] yum install httpd -y

2.修改keepalived配置文件

master配置:

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
        1638581716@qq.com 
        #keepalived 发生故障切换时邮件发送的目标邮箱,可以按行区分写多个
   }
   notification_email_from keepalived@timinglee.org #发邮件的地址
   smtp_server 127.0.0.1 #邮件服务器地址
   smtp_connect_timeout 30 #邮件服务器连接timeout
   router_id ka1.timinglee.org 
   #每个keepalived主机唯一标识,建议使用当前主机名,但多节点
重名不影响
   vrrp_skip_check_adv_addr #对所有通告报文都检查
   #vrrp_strict  #添加此选项无法访问vip,可以用nft list ruleset查看
   vrrp_garp_interval 0 #报文发送延迟,0表示不延迟
   vrrp_gna_interval 0 #消息发送延迟
   vrrp_mcast_group4 224.0.0.18 #指定组播IP地址范围:
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0 #绑定为当前虚拟路由器使用的物理接口,可以和VIP不在一个网卡
    virtual_router_id 100 #每个虚拟路由器唯一标识,范围:0-255,每个虚拟路由器此值必须唯一
    priority 100 #当前物理节点在此虚拟路由器的优先级,范围:1-254
    advert_int 1 #vrrp通告的时间间隔,默认1s
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress { #虚拟IP,生产环境可能指定上百个IP地址
        172.25.254.100/24 dev eth0 label eth0:1  #指定VIP,不指定网卡,默认为eth0
    }
}

slave配置:

[root@ka2 ~] vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
        1638581716@qq.com
   }
   notification_email_from keepalived@timinglee.org
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka1.timinglee.org
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   vrrp_mcast_group4 224.0.0.18
}

vrrp_instance VI_1 {
    state BACKUP #此实验为单主实验,采用默认抢占模式,只有一个master
    interface eth0
    virtual_router_id 100 #与master id 相同,相同的id管理同一个虚拟路由
    priority 80 #低优先级
    advert_int 1
    #preempt_delay 5s
    #nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
}

效果显示

#重启keepalived服务后检测
[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 18207  bytes 1283963 (1.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 32855  bytes 2470056 (2.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet) #ka1上出现vip
        
[root@ka1 ~] tcpdump -i eth0 -nn host 224.0.0.18 #通过抓包查看组播
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:03:03.286567 IP 172.25.254.10 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20

如果ka1关闭了keepalived.service,tcpdump则为20 > 224.0.0.18,再打开则再从10 > 224.0.0.18。因为默认为抢占模式,ka1为master,高优先级,打开后抢占vip

自定义日志路径

[root@ka1 ~] vim /etc/sysconfig/keepalived 
  KEEPALIVED_OPTIONS="-D -S 6" #范围为0-7

[root@ka1 ~] vim /etc/rsyslog.conf
# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log
local6.*                                                /var/log/keepalived.log
[root@ka1 ~] systemctl restart rsyslog.service 
[root@ka1 ~] systemctl restart keepalived.service 
[root@ka1 ~] ll /var/log/keepalived.log 
-rw-------. 1 root root 101381 Aug 12 14:58 /var/log/keepalived.log
#自动生成keepalived.log文件

实现独立子配置文件

当生产环境复杂时, /etc/keepalived/keepalived.conf 文件中内容过多,不易管理

将不同集群的配置,比如:不同集群的VIP配置放在独立的子配置文件中利用include 指令可以实现包含子配置文件

[root@ka1 ~] mkdir -p /etc/keepalived/conf.d 创建放子配置文件的目录
[root@ka1 ~] vim /etc/keepalived/keepalived.conf
#vrrp_instance VI_1 {  #讲以下配置注释并写入子配置文件中,以达到在主配置文件中相同的效果
#    state MASTER
#    interface eth0
#    virtual_router_id 100
#    priority 100
#    advert_int 1
   #preempt_delay 5s
   # nopreempt
#    authentication {
#        auth_type PASS
#        auth_pass 1111
#    }
#    virtual_ipaddress {
#        172.25.254.100/24 dev eth0 label eth0:1
#    }
#    unicast_src_ip 172.25.254.10
#    unicast_peer {
#        172.25.254.20
#   }
#}
include "/etc/keepalived/conf.d/*.conf" #相关子配置文件
[root@ka1 ~] systemctl restart keepalived.service
[root@ka1 ~] cat /etc/keepalived/conf.d/172.25.254.100.conf 
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
}

效果展示:

[root@ka1 ~]# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 2195  bytes 161604 (157.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4582  bytes 324321 (316.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
#在ka1中仍有vip

Keepalived常用模式

1.抢占模式(默认)

默认为抢占模式preempt,即当高优先级的主机恢复在线后,会抢占低先级的主机的master角色,这样会使vip在KA主机中来回漂移,造成网络抖动

具体步骤见单主架构实验

2.非抢占模式

非抢占模式 nopreempt ,即高优先级主机恢复后,并不会抢占低优先级主机的vip

非抢占模块下,如果原主机down机, VIP迁移至的新主机, 后续再发生down时,才会将VIP迁移回原主机

                              ####注意####
#在配置实验之前需要将global_defs中的vrrp_strict注释掉,添加此选项无法访问vip
                              ############
#ka1主机配置
vrrp_instance VI_1 {
    state BACKUP #两台主机均设置为backup
    interface eth0
    virtual_router_id 100
    priority 100 #高优先级
    advert_int 1
   #preempt_delay 5s
    nopreempt #使用非抢占模式
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.10
    unicast_peer {
        172.25.254.20
   }
}

#ka2主机配置
vrrp_instance VI_1 {
    state BACKUP #使用backup
    interface eth0
    virtual_router_id 100
    priority 80 #低优先级
    advert_int 1
    #preempt_delay 5s
    nopreempt #启用非抢占模式
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
}

可通过ka1上的keepalived服务开启和关闭检测vip位置和是否被抢占

3.延时抢占模式

抢占延迟模式,即优先级高的主机恢复后,不会立即抢回VIP,而是延迟一段时间(默认300s)再抢回VIP

                              ####注意####
#在配置实验之前需要将global_defs中的vrrp_strict注释掉,添加此选项无法访问vip
                              ############
#ka1主机配置
vrrp_instance VI_1 {
    state BACKUP #两台主机均设置为backup
    interface eth0
    virtual_router_id 100
    priority 100 #高优先级
    advert_int 1
    preempt_delay 5s #抢占延时5s
   #nopreempt 
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.10
    unicast_peer {
        172.25.254.20
   }
}
#ka2主机配置
vrrp_instance VI_1 {
    state BACKUP #使用backup
    interface eth0
    virtual_router_id 100
    priority 80 #低优先级
    advert_int 1
    preempt_delay 5s #抢占延时5s
    #nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
}

可通过ka1上的keepalived服务开启和关闭检测vip位置和是否进行了延时抢占

VIP单播配置

默认keepalived主机之间利用多播相互通告消息,会造成网络拥塞,可以替换成单播,减少网络流量

#master配置
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf 
global_defs {
   notification_email {
        1638581716@qq.com
   }
   notification_email_from keepalived@timinglee.org
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka1.timinglee.org
   vrrp_skip_check_adv_addr
   #vrrp_strict  #参数开启会与vip单播产生冲突,服务无法启动
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   vrrp_mcast_group4 224.0.0.18
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
   #preempt_delay 5s
   # nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.10 #本机ip
    unicast_peer {
        172.25.254.20 #指向的对方主机ip
   }
}

#backup配置
[root@ka2 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
        1638581716@qq.com
   }
   notification_email_from keepalived@timinglee.org
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id ka1.timinglee.org
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   vrrp_mcast_group4 224.0.0.18
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 100
    priority 80
    advert_int 1
    #preempt_delay 5s
    #nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.20 #本机ip
    unicast_peer {
        172.25.254.10 #指定对方主机ip
    }
}

效果展示

#当ka1,ka2都开启服务时,vip在ka1上
[root@ka1 ~] tcpdump -i eth0 -nn src host 172.25.254.10 and dst 172.25.254.20
#注意host后面为本机到指定的主机
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:20:56.107869 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
22:20:57.108921 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
#当ka1关闭服务时,vip转到ka2上
[root@ka2 ~] tcpdump -i eth0 -nn src host 172.25.254.20 and dst 172.25.254.10
22:24:10.695469 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
22:24:11.696617 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
22:24:12.697912 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
#此时再将ka1服务打开,vip又回到ka1上
#ka1显示
22:25:55.812012 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
22:25:56.812921 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
#ka2显示
22:24:52.751748 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
22:24:56.763268 ARP, Request who-has 172.25.254.10 tell 172.25.254.20, length 28
22:25:04.767058 ARP, Reply 172.25.254.20 is-at 00:0c:29:db:c9:5c, length 28
22:25:44.814841 ARP, Reply 172.25.254.20 is-at 00:0c:29:db:c9:5c, length 28

Keepalived通知脚本配置

当keepalived的状态变化时,可以自动触发脚本的执行,比如:发邮件通知用户

默认以用户keepalived_script身份执行脚本

1.创建通知脚本

[root@ka1 ~] vim /etc/keepalived/mail.sh
#!/bin/bash
mail_dst="775762675@qq.com"
send_message()
{
 mail_sub="$HOSTNAME to be $1 vip move"
 mail_msg="'date +%F\ %T': vrrp move $HOSTNAME chage $1"
 echo $mail_msg | mail -s "$mail_sub" $mail_dst
}

case $1 in
 	master)
	send_message master
 	;;
 	backup)
 	send_message backup
 	;;
 	fauil)
 	send_message fault
 	;;
 	*)
 	;;
esac

2.安装邮箱发送工具并进行配置

[root@ka1 ~] yum install mailx -y
[root@ka1 ~] vim /etc/mail.rc #在末尾添加
# For Linux and BSD, this should be set.
set bsdcompat
set from=775762675@qq.com
set smtp=smtp.qq.com #以qq邮箱为例子,需要打开qq邮箱的权限,并生成码
set smtp-auth-user=775762675@qq.com
set smtp-auth-password=njgtlyplsnktbbbb #密码为qq邮箱生成的码
set smtp-auth=login
set ssl-verify=ignore

效果演示:

[root@ka1 ~] echo test message | mail -s test 775762675@qq.com
#发送测试邮箱

实现keepalived双主模式

单主模式下,同一时间只有一个keepalived对外提供服务,一台主机繁忙另一台主机空闲。双主模式则是解决效率低下的方法。

双主模式:将两个或以上的vip运行在不同的keepalived服务器,以实现服务器并提供web访问,提高服务器利用率

#ka1配置
vrrp_instance VI_1 { #虚拟路由1
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
   #preempt_delay 5s
   # nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.10
    unicast_peer {
        172.25.254.20
   }
   track_script {
        check_haproxy
   }
}
vrrp_instance VI_2 { #虚拟路由2
    state BACKUP #做为第二个虚拟路由的从
    interface eth0
    virtual_router_id 200 #虚拟路由的唯一标识,与虚拟路由1要不同
    priority 80 
    advert_int 1
   #preempt_delay 5s
   # nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.200/24 dev eth0 label eth0:2 #vip为200,端口为2
    }
    unicast_src_ip 172.25.254.10 #单播仍为从10(本机)发往20(目标主机)
    unicast_peer {
        172.25.254.20
   }
}

#ka2配置
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 100
    priority 80
    advert_int 1
    #preempt_delay 5s
    #nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.20
    unicast_peer {
        172.25.254.10
    }
    track_script {
        check_file
    }
}

vrrp_instance VI_2 {
    state MASTER
    interface eth0
    virtual_router_id 200
    priority 100
    advert_int 1
    #preempt_delay 5s
    #nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.200/24 dev eth0 label eth0:2
    }
    unicast_src_ip 172.25.254.20
    unicast_peer {
        172.25.254.10
    }
}

效果演示:

[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 5255  bytes 450020 (439.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 364887  bytes 27000285 (25.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 6293  bytes 323110 (315.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6293  bytes 323110 (315.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@ka2 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.20  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fedb:c95c  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:db:c9:5c  txqueuelen 1000  (Ethernet)
        RX packets 4911  bytes 384192 (375.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4068  bytes 350877 (342.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.200  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:db:c9:5c  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 6296  bytes 323206 (315.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6296  bytes 323206 (315.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

三.keepalived和lvs结合实现高可用负载均衡集群

virtual server 的定义格式

virtual_server IP port #定义虚拟主机ip地址和端口
virtual_server fwmark int #ipvs的防火墙打标记,实现基于防火墙的负载均衡集群
virtual_server group string #使用虚拟服务器组

实验步骤

1.虚拟服务器配置
virtual_server 172.25.254.100 80 { #vip及端口
    delay_loop 6 #后端服务器检查的时间间隔
    lb_algo wrr #调度方法
    lb_kind DR #集群类型:NAT|DR|TUN(nat模式,直连路由模式,隧道模式)
    protocol TCP #指定服务协议为tcp,一般情况下为tcp
#相当于在设置:
#ipvsadm -A -t 172.25.254.100:80 -s wrr

    real_server 172.25.254.110 80 { #后端主机的ip和端口
        weight 1
        HTTP_GET { #应用层监控,定义对后端主机做到检测动作
            url {
              path / #定义访问的url
              status_code 200 #判断检测机制为健康状态的响应码,一般为200
            }
            connect_timeout 3 #客户端请求超时时长
            nb_get_retry 2 #重试次数
            delay_before_retry 2 #重启之前的延迟时长
        }
    }
#相当于ipvsadm -a -t 172.25.254.100:80 -r 172.25.254.110:80 -g -w 1
    real_server 172.25.254.120 80 {
        weight 1
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 3
            nb_get_retry 2
            delay_before_retry 2
        }
    }
}
#相当于ipvsadm -a -t 172.25.254.100:80 -r 172.25.254.120:80 -g
2.lvs环境配置
#两主机配置vip
[root@rs1 ~] ip a a 172.25.254.100/32 dev lo 
#vip设定为在那台虚拟路由上实现ipvs就使用那台虚拟路由生成的vip
[root@rs2 ~] ip a a 172.25.254.100/32 dev lo
3.arp广播设定,解决vip响应问题
[root@rs1 ~] vim /etc/sysctl.d/arp.conf #写入子配置文件中,也能写入sysctl.conf中
[root@rs1 ~] cat /etc/sysctl.d/arp.conf 
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
[root@rs1 ~] sysctl --system
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
* Applying /etc/sysctl.conf ...
[root@rs1 ~] scp /etc/sysctl.d/arp.conf root@172.25.254.120:/etc/sysctl.d/arp.conf 
#将文件复制到120上
[root@rs2 ~] sysctl --system

4.检测ipvs策略
[root@ka1 ~] yum install ipvsadm -y
[root@ka1 ~] ipvsadm-Ln #重启keepalived服务后能看见ipvs策略
-bash: ipvsadm-Ln: command not found
[root@ka1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.254.100:80 wrr
  -> 172.25.254.110:80            Route   1      0          0         
  -> 172.25.254.120:80            Route   1      0          0       
5.效果检测:
[root@servera ~] for i in {1..6}
> do
> curl 172.25.254.100
> done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120

实验实现的作用

lvs本身不提供后端检查,通过与keepalived结合,当ipvs策略设置完成,若有一台主机服务关闭,通过keepalived的检测可以直接将关闭的主机从ipvs的策略中剔除,主机服务上线后又会将主机自动加载到策略中

[root@ka1 ~] watch -n 1 ipvsadm -Ln

在这里插入图片描述

#关闭一台主机的http服务
[root@rs1 ~] systemctl stop httpd.service 

在这里插入图片描述

[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
#再开启服务
[root@rs1 ~]# systemctl restart httpd.service

在这里插入图片描述

[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120

当ka2也进行了相关配置后,虽然也有ipvs策略但是不生效,因为vip不在ka2上

[root@ka2 ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.254.100:80 wrr
  -> 172.25.254.110:80            Route   1      0          0         
  -> 172.25.254.120:80            Route   1      0          0       

但当ka1关闭keepalived服务后,ka2获得vip,ka2上的策略就能生效,客户端的检测结果不会受到影响。

[root@ka1 ~] systemctl stop keepalived.service 
[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 1697761  bytes 124362381 (118.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3741326  bytes 264319078 (252.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 23187  bytes 1187255 (1.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 23187  bytes 1187255 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[root@ka2 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.20  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fedb:c95c  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:db:c9:5c  txqueuelen 1000  (Ethernet)
        RX packets 23305  bytes 1682916 (1.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25000  bytes 1917704 (1.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:db:c9:5c  txqueuelen 1000  (Ethernet)

[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110

但因为默认设定为抢占模式,在客户端的调度可能会受影响,因为两台主机的ipvs策略是各自的,调度会因vip飘逸而改变,在企业中一般采用非抢占模式或者延迟抢占

四.keepalived与haproxy的结合实现高可用负载均衡集群

VRRP Script

keepalived可以通过VRRP Script技术,调用外部的辅助脚本进行资源监控,并根据监控的结果实现优先动态的调度(修改优先级),从而实现其他应用的高可用性功能

vrrp script配置
#先定义
vrrp_script check_haproxy { 		#定义一个检测脚本
   script "/etc/keepalived/test.sh" #SHELL命令或编写的脚本的路径
   interval 1  						#脚本执行间隔
   weight -30 						#权重,默认为0,范围为<-254..254>
   			  						#当设值为负时,相当于减去此大小的权重
   			  						#当脚本返回值为非0的时候才生效
   fall 2  							#执行脚本连续失败2次则视为失败
   rise 2							#执行脚本连续2次成功则视为成功
   timeout  2						#
}

#后调用
 track_script {
        check_haproxy 				#调用定义的检测脚本名称		
   }

实验步骤:
1.编写脚本
[root@ka1 ~] vim /etc/keepalived/test.sh
#!/bin/bash
[ ! -f /mnt/haha ]
ech0 $? #便于观测写上,如要写入keepalived中的检测检测脚本中不能写,否则返回值都为0
#当有/mnt/haha这个文件时,返回值为非0,没有文件时返回值为0
[root@ka1 ~] touch /mnt/haha
[root@ka1 ~] bash /etc/keepalived/test.sh 
1
[root@ka1 ~] rm -rf /mnt/haha 
[root@ka1 ~] bash /etc/keepalived/test.sh 
0
2.配置keepalived的检测脚本
vrrp_script check_haproxy {
   script "/etc/keepalived/test.sh" #注意与编写的脚本地址一致
   interval 1
   weight -30 
   #当/mnt/haha创建后,返回值为非0,ka1的权重减去30,小于ka2,vip移值ka2
   fall 2
   rise 2
   timeout  2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
   #preempt_delay 5s
   # nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.10
    unicast_peer {
        172.25.254.20
   }
   track_script {
        check_haproxy
   }
}
3.效果展示
[root@ka1 ~] touch /mnt/haha
[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 1702602  bytes 124705117 (118.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3746687  bytes 264789670 (252.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 23439  bytes 1206467 (1.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 23439  bytes 1206467 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@ka1 ~] rm -rf /mnt/haha
[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 1708819  bytes 125096721 (119.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3751282  bytes 265123270 (252.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)


实现haproxy高可用

1.安装haproxy

[root@ka1 ~] yum install haproxy -y
[root@ka2 ~] yum install haproxy -y

2.在ka1与ka2上启用内核参数

[root@ka1 ~] vim /etc/sysctl.conf
[root@ka1 ~] cat /etc/sysctl.conf 
net.ipv4.ip_nonlocal_bind=1 #使haproxy能进行远程监听
[root@ka1 ~] sysctl -p
net.ipv4.ip_nonlocal_bind = 1
#vip会因为检测脚本而发生偏移,此配置是为了让haproxy本地没有这个IP也能正常启动
[root@ka2 ~] vim /etc/sysctl.conf
[root@ka2 ~] cat /etc/sysctl.conf 
net.ipv4.ip_nonlocal_bind=1 #使haproxy能进行远程监听
[root@ka2 ~] sysctl -p
net.ipv4.ip_nonlocal_bind = 1

3.配置haproxy

[root@ka1 ~] vim /etc/haproxy/haproxy.cfg
listen webcluster
    bind 172.25.254.100:80
    mode http
    balance roundrobin
    server web1 172.25.254.110:80 check inter 2 fall 3 rise 5
    server web2 172.25.254.120:80 check inter 2 fall 3 rise 5
[root@ka1 ~] systemctl restart haproxy.service    
[root@ka1 ~] netstat -antlupe | grep haproxy #查看端口是否开启
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      0          2296855    29484/haproxy       
tcp        0      0 172.25.254.100:80       0.0.0.0:*               LISTEN      0          2296857    29484/haproxy       
[root@ka2 ~] vim /etc/haproxy/haproxy.cfg 
listen webcluster
    bind 172.25.254.100:80
    mode http
    balance roundrobin
    server web1 172.25.254.110:80 check
    server web2 172.25.254.120:80 check
[root@ka2 ~] systemctl restart haproxy.service

效果展示

[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
#注意,haproxy和lvs在同一虚拟路由上不能混合使用,完成此实验之前需要将lvs实验的配置还原,可以使用双主机模式将二者写在不同的虚拟路由上,使用不同的vip

当ka1关闭haproxy服务时,拥有vip的主机关闭了haproxy服务,导致客户机上链接被拒绝

[root@ka1 ~] systemctl stop haproxy.service 
[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 2211115  bytes 162252499 (154.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4776951  bytes 337045610 (321.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused

可以通过killall命令查看haproxy的状态

[root@ka1 ~] killall -0 haproxy #查看已经关闭了haproxy服务的主机
haproxy: no process found #报错说明此时的haproxy的返回值为非0
[root@ka1 ~] echo $? 
1
[root@ka2 ~] killall -0 haproxy #查看未关闭haproxy服务的主机
[root@ka2 ~] echo $?
0

4.编写脚本

[root@ka1 ~]# vim /etc/keepalived/test.sh 
[root@ka1 ~]# cat /etc/keepalived/test.sh 
#!/bin/bash
killall -0 haproxy
[root@ka2 ~]# vim /etc/keepalived/test.sh 
[root@ka2 ~]# cat /etc/keepalived/test.sh 
#!/bin/bash
killall -0 haproxy

5.使用vrrp script并编写检测脚本

vrrp_script check_haproxy {
   script "/etc/keepalived/test.sh"
   interval 1
   weight -30
   fall 2
   rise 2
   timeout  2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
   #preempt_delay 5s
   # nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1
    }
    unicast_src_ip 172.25.254.10
    unicast_peer {
        172.25.254.20
   }
   track_script {
        check_haproxy
   }
}
#ka2也需要配置,因为可能会涉及到抢占模式
[root@ka1 ~] systemctl restart keepalived.service

效果展示:

[root@ka1 ~] systemctl restart haproxy.service 
[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 2230513  bytes 163680073 (156.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4814430  bytes 339689220 (323.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120

当ka1上关闭haproxy服务

[root@ka1 ~] systemctl stop haproxy.service 
[root@ka1 ~] ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.254.10  netmask 255.255.255.0  broadcast 172.25.254.255
        inet6 fe80::20c:29ff:fec8:f6c4  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c8:f6:c4  txqueuelen 1000  (Ethernet)
        RX packets 2254543  bytes 165457815 (157.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4862438  bytes 343051166 (327.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 28777  bytes 1496893 (1.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28777  bytes 1496893 (1.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120

  • 11
    点赞
  • 21
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值