高可用集群KEEPALIVED
一.高可用集群
1.1 集群类型
-
LB:Load Balance 负载均衡
- LVS/HAProxy/nginx(http/upstream, stream/upstream)
-
HA:High Availability 高可用集群
- 数据库、Redis
-
SPoF: Single Point of Failure,解决单点故障
- HPC:High Performance Computing 高性能集群
1.2 系统可用性
SLA:Service-Level Agreement 服务等级协议(提供服务的企业与客户之间就服务的品质、水准、性能
等方面所达成的双方共同认可的协议或契约)
A = MTBF / (MTBF+MTTR)
指标 :99.9%, 99.99%, 99.999%,99.9999%
1.3 系统故障
硬件故障:设计缺陷、wear out(损耗)、非人为不可抗拒因素
软件故障:设计缺陷 bug
1.4 实现高可用
提升系统高用性的解决方案:降低MTTR- Mean Time To Repair(平均故障时间)
解决方案:建立冗余机制
- active/passive 主/备
- active/active 双主
- active --> HEARTBEAT --> passive
- active <–> HEARTBEAT <–> active
1.5.VRRP:Virtual Router Redundancy Protocol
虚拟路由冗余协议,解决静态网关单点风险
- 物理层:路由器、三层交换机
- 软件层:keepalived
1.5.1 VRRP 相关术语
- 虚拟路由器:Virtual Router
- 虚拟路由器标识:VRID(0-255),唯一标识虚拟路由器
- VIP:Virtual IP
- VMAC:Virutal MAC (00-00-5e-00-01-VRID)
- 物理路由器:
- master:主设备
- backup:备用设备
- priority:优先级
1.5.2 VRRP 相关技术
通告:心跳,优先级等;周期性
工作方式:抢占式,非抢占式
安全认证:
- 无认证
- 简单字符认证:预共享密钥
- MD5
工作模式:
- 主/备:单虚拟路由器
- 主/主:主/备(虚拟路由器1),备/主(虚拟路由器2)
二.Keepalived基础
实现master/slave的keepalived单主架构实验
实验准备:准备四台rhel7.9虚拟机
根据拓扑图配置ip,并关闭防火墙和selinux
1.安装keepalived,http
[root@ka1 ~] yum install keepalived.x86_64 -y
[root@ka2 ~] yum install keepalived.x86_64 -y
[root@rs1 ~] yum install httpd -y
[root@rs2 ~] yum install httpd -y
2.修改keepalived配置文件
master配置:
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
1638581716@qq.com
#keepalived 发生故障切换时邮件发送的目标邮箱,可以按行区分写多个
}
notification_email_from keepalived@timinglee.org #发邮件的地址
smtp_server 127.0.0.1 #邮件服务器地址
smtp_connect_timeout 30 #邮件服务器连接timeout
router_id ka1.timinglee.org
#每个keepalived主机唯一标识,建议使用当前主机名,但多节点
重名不影响
vrrp_skip_check_adv_addr #对所有通告报文都检查
#vrrp_strict #添加此选项无法访问vip,可以用nft list ruleset查看
vrrp_garp_interval 0 #报文发送延迟,0表示不延迟
vrrp_gna_interval 0 #消息发送延迟
vrrp_mcast_group4 224.0.0.18 #指定组播IP地址范围:
}
vrrp_instance VI_1 {
state MASTER
interface eth0 #绑定为当前虚拟路由器使用的物理接口,可以和VIP不在一个网卡
virtual_router_id 100 #每个虚拟路由器唯一标识,范围:0-255,每个虚拟路由器此值必须唯一
priority 100 #当前物理节点在此虚拟路由器的优先级,范围:1-254
advert_int 1 #vrrp通告的时间间隔,默认1s
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress { #虚拟IP,生产环境可能指定上百个IP地址
172.25.254.100/24 dev eth0 label eth0:1 #指定VIP,不指定网卡,默认为eth0
}
}
slave配置:
[root@ka2 ~] vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
1638581716@qq.com
}
notification_email_from keepalived@timinglee.org
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.timinglee.org
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}
vrrp_instance VI_1 {
state BACKUP #此实验为单主实验,采用默认抢占模式,只有一个master
interface eth0
virtual_router_id 100 #与master id 相同,相同的id管理同一个虚拟路由
priority 80 #低优先级
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
}
效果显示
#重启keepalived服务后检测
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 18207 bytes 1283963 (1.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 32855 bytes 2470056 (2.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet) #ka1上出现vip
[root@ka1 ~] tcpdump -i eth0 -nn host 224.0.0.18 #通过抓包查看组播
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:03:03.286567 IP 172.25.254.10 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
如果ka1关闭了keepalived.service,tcpdump则为20 > 224.0.0.18,再打开则再从10 > 224.0.0.18。因为默认为抢占模式,ka1为master,高优先级,打开后抢占vip
自定义日志路径
[root@ka1 ~] vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 6" #范围为0-7
[root@ka1 ~] vim /etc/rsyslog.conf
# Save boot messages also to boot.log
local7.* /var/log/boot.log
local6.* /var/log/keepalived.log
[root@ka1 ~] systemctl restart rsyslog.service
[root@ka1 ~] systemctl restart keepalived.service
[root@ka1 ~] ll /var/log/keepalived.log
-rw-------. 1 root root 101381 Aug 12 14:58 /var/log/keepalived.log
#自动生成keepalived.log文件
实现独立子配置文件
当生产环境复杂时, /etc/keepalived/keepalived.conf 文件中内容过多,不易管理
将不同集群的配置,比如:不同集群的VIP配置放在独立的子配置文件中利用include 指令可以实现包含子配置文件
[root@ka1 ~] mkdir -p /etc/keepalived/conf.d 创建放子配置文件的目录
[root@ka1 ~] vim /etc/keepalived/keepalived.conf
#vrrp_instance VI_1 { #讲以下配置注释并写入子配置文件中,以达到在主配置文件中相同的效果
# state MASTER
# interface eth0
# virtual_router_id 100
# priority 100
# advert_int 1
#preempt_delay 5s
# nopreempt
# authentication {
# auth_type PASS
# auth_pass 1111
# }
# virtual_ipaddress {
# 172.25.254.100/24 dev eth0 label eth0:1
# }
# unicast_src_ip 172.25.254.10
# unicast_peer {
# 172.25.254.20
# }
#}
include "/etc/keepalived/conf.d/*.conf" #相关子配置文件
[root@ka1 ~] systemctl restart keepalived.service
[root@ka1 ~] cat /etc/keepalived/conf.d/172.25.254.100.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
}
效果展示:
[root@ka1 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 2195 bytes 161604 (157.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4582 bytes 324321 (316.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
#在ka1中仍有vip
Keepalived常用模式
1.抢占模式(默认)
默认为抢占模式preempt,即当高优先级的主机恢复在线后,会抢占低先级的主机的master角色,这样会使vip在KA主机中来回漂移,造成网络抖动
具体步骤见单主架构实验
2.非抢占模式
非抢占模式 nopreempt ,即高优先级主机恢复后,并不会抢占低优先级主机的vip
非抢占模块下,如果原主机down机, VIP迁移至的新主机, 后续再发生down时,才会将VIP迁移回原主机
####注意####
#在配置实验之前需要将global_defs中的vrrp_strict注释掉,添加此选项无法访问vip
############
#ka1主机配置
vrrp_instance VI_1 {
state BACKUP #两台主机均设置为backup
interface eth0
virtual_router_id 100
priority 100 #高优先级
advert_int 1
#preempt_delay 5s
nopreempt #使用非抢占模式
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.10
unicast_peer {
172.25.254.20
}
}
#ka2主机配置
vrrp_instance VI_1 {
state BACKUP #使用backup
interface eth0
virtual_router_id 100
priority 80 #低优先级
advert_int 1
#preempt_delay 5s
nopreempt #启用非抢占模式
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
}
可通过ka1上的keepalived服务开启和关闭检测vip位置和是否被抢占
3.延时抢占模式
抢占延迟模式,即优先级高的主机恢复后,不会立即抢回VIP,而是延迟一段时间(默认300s)再抢回VIP
####注意####
#在配置实验之前需要将global_defs中的vrrp_strict注释掉,添加此选项无法访问vip
############
#ka1主机配置
vrrp_instance VI_1 {
state BACKUP #两台主机均设置为backup
interface eth0
virtual_router_id 100
priority 100 #高优先级
advert_int 1
preempt_delay 5s #抢占延时5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.10
unicast_peer {
172.25.254.20
}
}
#ka2主机配置
vrrp_instance VI_1 {
state BACKUP #使用backup
interface eth0
virtual_router_id 100
priority 80 #低优先级
advert_int 1
preempt_delay 5s #抢占延时5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
}
可通过ka1上的keepalived服务开启和关闭检测vip位置和是否进行了延时抢占
VIP单播配置
默认keepalived主机之间利用多播相互通告消息,会造成网络拥塞,可以替换成单播,减少网络流量
#master配置
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
1638581716@qq.com
}
notification_email_from keepalived@timinglee.org
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.timinglee.org
vrrp_skip_check_adv_addr
#vrrp_strict #参数开启会与vip单播产生冲突,服务无法启动
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
#preempt_delay 5s
# nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.10 #本机ip
unicast_peer {
172.25.254.20 #指向的对方主机ip
}
}
#backup配置
[root@ka2 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
1638581716@qq.com
}
notification_email_from keepalived@timinglee.org
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ka1.timinglee.org
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 80
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.20 #本机ip
unicast_peer {
172.25.254.10 #指定对方主机ip
}
}
效果展示
#当ka1,ka2都开启服务时,vip在ka1上
[root@ka1 ~] tcpdump -i eth0 -nn src host 172.25.254.10 and dst 172.25.254.20
#注意host后面为本机到指定的主机
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:20:56.107869 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
22:20:57.108921 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
#当ka1关闭服务时,vip转到ka2上
[root@ka2 ~] tcpdump -i eth0 -nn src host 172.25.254.20 and dst 172.25.254.10
22:24:10.695469 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
22:24:11.696617 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
22:24:12.697912 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
#此时再将ka1服务打开,vip又回到ka1上
#ka1显示
22:25:55.812012 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
22:25:56.812921 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
#ka2显示
22:24:52.751748 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20
22:24:56.763268 ARP, Request who-has 172.25.254.10 tell 172.25.254.20, length 28
22:25:04.767058 ARP, Reply 172.25.254.20 is-at 00:0c:29:db:c9:5c, length 28
22:25:44.814841 ARP, Reply 172.25.254.20 is-at 00:0c:29:db:c9:5c, length 28
Keepalived通知脚本配置
当keepalived的状态变化时,可以自动触发脚本的执行,比如:发邮件通知用户
默认以用户keepalived_script身份执行脚本
1.创建通知脚本
[root@ka1 ~] vim /etc/keepalived/mail.sh
#!/bin/bash
mail_dst="775762675@qq.com"
send_message()
{
mail_sub="$HOSTNAME to be $1 vip move"
mail_msg="'date +%F\ %T': vrrp move $HOSTNAME chage $1"
echo $mail_msg | mail -s "$mail_sub" $mail_dst
}
case $1 in
master)
send_message master
;;
backup)
send_message backup
;;
fauil)
send_message fault
;;
*)
;;
esac
2.安装邮箱发送工具并进行配置
[root@ka1 ~] yum install mailx -y
[root@ka1 ~] vim /etc/mail.rc #在末尾添加
# For Linux and BSD, this should be set.
set bsdcompat
set from=775762675@qq.com
set smtp=smtp.qq.com #以qq邮箱为例子,需要打开qq邮箱的权限,并生成码
set smtp-auth-user=775762675@qq.com
set smtp-auth-password=njgtlyplsnktbbbb #密码为qq邮箱生成的码
set smtp-auth=login
set ssl-verify=ignore
效果演示:
[root@ka1 ~] echo test message | mail -s test 775762675@qq.com
#发送测试邮箱
实现keepalived双主模式
单主模式下,同一时间只有一个keepalived对外提供服务,一台主机繁忙另一台主机空闲。双主模式则是解决效率低下的方法。
双主模式:将两个或以上的vip运行在不同的keepalived服务器,以实现服务器并提供web访问,提高服务器利用率
#ka1配置
vrrp_instance VI_1 { #虚拟路由1
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
#preempt_delay 5s
# nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.10
unicast_peer {
172.25.254.20
}
track_script {
check_haproxy
}
}
vrrp_instance VI_2 { #虚拟路由2
state BACKUP #做为第二个虚拟路由的从
interface eth0
virtual_router_id 200 #虚拟路由的唯一标识,与虚拟路由1要不同
priority 80
advert_int 1
#preempt_delay 5s
# nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.200/24 dev eth0 label eth0:2 #vip为200,端口为2
}
unicast_src_ip 172.25.254.10 #单播仍为从10(本机)发往20(目标主机)
unicast_peer {
172.25.254.20
}
}
#ka2配置
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 80
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.20
unicast_peer {
172.25.254.10
}
track_script {
check_file
}
}
vrrp_instance VI_2 {
state MASTER
interface eth0
virtual_router_id 200
priority 100
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.200/24 dev eth0 label eth0:2
}
unicast_src_ip 172.25.254.20
unicast_peer {
172.25.254.10
}
}
效果演示:
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 5255 bytes 450020 (439.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 364887 bytes 27000285 (25.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 6293 bytes 323110 (315.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6293 bytes 323110 (315.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ka2 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.20 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fedb:c95c prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:db:c9:5c txqueuelen 1000 (Ethernet)
RX packets 4911 bytes 384192 (375.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4068 bytes 350877 (342.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.200 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:db:c9:5c txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 6296 bytes 323206 (315.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6296 bytes 323206 (315.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
三.keepalived和lvs结合实现高可用负载均衡集群
virtual server 的定义格式
virtual_server IP port #定义虚拟主机ip地址和端口
virtual_server fwmark int #ipvs的防火墙打标记,实现基于防火墙的负载均衡集群
virtual_server group string #使用虚拟服务器组
实验步骤
1.虚拟服务器配置
virtual_server 172.25.254.100 80 { #vip及端口
delay_loop 6 #后端服务器检查的时间间隔
lb_algo wrr #调度方法
lb_kind DR #集群类型:NAT|DR|TUN(nat模式,直连路由模式,隧道模式)
protocol TCP #指定服务协议为tcp,一般情况下为tcp
#相当于在设置:
#ipvsadm -A -t 172.25.254.100:80 -s wrr
real_server 172.25.254.110 80 { #后端主机的ip和端口
weight 1
HTTP_GET { #应用层监控,定义对后端主机做到检测动作
url {
path / #定义访问的url
status_code 200 #判断检测机制为健康状态的响应码,一般为200
}
connect_timeout 3 #客户端请求超时时长
nb_get_retry 2 #重试次数
delay_before_retry 2 #重启之前的延迟时长
}
}
#相当于ipvsadm -a -t 172.25.254.100:80 -r 172.25.254.110:80 -g -w 1
real_server 172.25.254.120 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}
}
#相当于ipvsadm -a -t 172.25.254.100:80 -r 172.25.254.120:80 -g
2.lvs环境配置
#两主机配置vip
[root@rs1 ~] ip a a 172.25.254.100/32 dev lo
#vip设定为在那台虚拟路由上实现ipvs就使用那台虚拟路由生成的vip
[root@rs2 ~] ip a a 172.25.254.100/32 dev lo
3.arp广播设定,解决vip响应问题
[root@rs1 ~] vim /etc/sysctl.d/arp.conf #写入子配置文件中,也能写入sysctl.conf中
[root@rs1 ~] cat /etc/sysctl.d/arp.conf
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
[root@rs1 ~] sysctl --system
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
* Applying /etc/sysctl.conf ...
[root@rs1 ~] scp /etc/sysctl.d/arp.conf root@172.25.254.120:/etc/sysctl.d/arp.conf
#将文件复制到120上
[root@rs2 ~] sysctl --system
4.检测ipvs策略
[root@ka1 ~] yum install ipvsadm -y
[root@ka1 ~] ipvsadm-Ln #重启keepalived服务后能看见ipvs策略
-bash: ipvsadm-Ln: command not found
[root@ka1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:80 wrr
-> 172.25.254.110:80 Route 1 0 0
-> 172.25.254.120:80 Route 1 0 0
5.效果检测:
[root@servera ~] for i in {1..6}
> do
> curl 172.25.254.100
> done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
实验实现的作用
lvs本身不提供后端检查,通过与keepalived结合,当ipvs策略设置完成,若有一台主机服务关闭,通过keepalived的检测可以直接将关闭的主机从ipvs的策略中剔除,主机服务上线后又会将主机自动加载到策略中
[root@ka1 ~] watch -n 1 ipvsadm -Ln
#关闭一台主机的http服务
[root@rs1 ~] systemctl stop httpd.service
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
rs2 - 172.25.254.120
#再开启服务
[root@rs1 ~]# systemctl restart httpd.service
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
当ka2也进行了相关配置后,虽然也有ipvs策略但是不生效,因为vip不在ka2上
[root@ka2 ~] ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:80 wrr
-> 172.25.254.110:80 Route 1 0 0
-> 172.25.254.120:80 Route 1 0 0
但当ka1关闭keepalived服务后,ka2获得vip,ka2上的策略就能生效,客户端的检测结果不会受到影响。
[root@ka1 ~] systemctl stop keepalived.service
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 1697761 bytes 124362381 (118.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3741326 bytes 264319078 (252.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 23187 bytes 1187255 (1.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23187 bytes 1187255 (1.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ka2 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.20 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fedb:c95c prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:db:c9:5c txqueuelen 1000 (Ethernet)
RX packets 23305 bytes 1682916 (1.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 25000 bytes 1917704 (1.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:db:c9:5c txqueuelen 1000 (Ethernet)
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
但因为默认设定为抢占模式,在客户端的调度可能会受影响,因为两台主机的ipvs策略是各自的,调度会因vip飘逸而改变,在企业中一般采用非抢占模式或者延迟抢占
四.keepalived与haproxy的结合实现高可用负载均衡集群
VRRP Script
keepalived可以通过VRRP Script技术,调用外部的辅助脚本进行资源监控,并根据监控的结果实现优先动态的调度(修改优先级),从而实现其他应用的高可用性功能
vrrp script配置
#先定义
vrrp_script check_haproxy { #定义一个检测脚本
script "/etc/keepalived/test.sh" #SHELL命令或编写的脚本的路径
interval 1 #脚本执行间隔
weight -30 #权重,默认为0,范围为<-254..254>
#当设值为负时,相当于减去此大小的权重
#当脚本返回值为非0的时候才生效
fall 2 #执行脚本连续失败2次则视为失败
rise 2 #执行脚本连续2次成功则视为成功
timeout 2 #
}
#后调用
track_script {
check_haproxy #调用定义的检测脚本名称
}
实验步骤:
1.编写脚本
[root@ka1 ~] vim /etc/keepalived/test.sh
#!/bin/bash
[ ! -f /mnt/haha ]
ech0 $? #便于观测写上,如要写入keepalived中的检测检测脚本中不能写,否则返回值都为0
#当有/mnt/haha这个文件时,返回值为非0,没有文件时返回值为0
[root@ka1 ~] touch /mnt/haha
[root@ka1 ~] bash /etc/keepalived/test.sh
1
[root@ka1 ~] rm -rf /mnt/haha
[root@ka1 ~] bash /etc/keepalived/test.sh
0
2.配置keepalived的检测脚本
vrrp_script check_haproxy {
script "/etc/keepalived/test.sh" #注意与编写的脚本地址一致
interval 1
weight -30
#当/mnt/haha创建后,返回值为非0,ka1的权重减去30,小于ka2,vip移值ka2
fall 2
rise 2
timeout 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
#preempt_delay 5s
# nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.10
unicast_peer {
172.25.254.20
}
track_script {
check_haproxy
}
}
3.效果展示
[root@ka1 ~] touch /mnt/haha
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 1702602 bytes 124705117 (118.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3746687 bytes 264789670 (252.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 23439 bytes 1206467 (1.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23439 bytes 1206467 (1.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ka1 ~] rm -rf /mnt/haha
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 1708819 bytes 125096721 (119.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3751282 bytes 265123270 (252.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
实现haproxy高可用
1.安装haproxy
[root@ka1 ~] yum install haproxy -y
[root@ka2 ~] yum install haproxy -y
2.在ka1与ka2上启用内核参数
[root@ka1 ~] vim /etc/sysctl.conf
[root@ka1 ~] cat /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1 #使haproxy能进行远程监听
[root@ka1 ~] sysctl -p
net.ipv4.ip_nonlocal_bind = 1
#vip会因为检测脚本而发生偏移,此配置是为了让haproxy本地没有这个IP也能正常启动
[root@ka2 ~] vim /etc/sysctl.conf
[root@ka2 ~] cat /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1 #使haproxy能进行远程监听
[root@ka2 ~] sysctl -p
net.ipv4.ip_nonlocal_bind = 1
3.配置haproxy
[root@ka1 ~] vim /etc/haproxy/haproxy.cfg
listen webcluster
bind 172.25.254.100:80
mode http
balance roundrobin
server web1 172.25.254.110:80 check inter 2 fall 3 rise 5
server web2 172.25.254.120:80 check inter 2 fall 3 rise 5
[root@ka1 ~] systemctl restart haproxy.service
[root@ka1 ~] netstat -antlupe | grep haproxy #查看端口是否开启
tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 0 2296855 29484/haproxy
tcp 0 0 172.25.254.100:80 0.0.0.0:* LISTEN 0 2296857 29484/haproxy
[root@ka2 ~] vim /etc/haproxy/haproxy.cfg
listen webcluster
bind 172.25.254.100:80
mode http
balance roundrobin
server web1 172.25.254.110:80 check
server web2 172.25.254.120:80 check
[root@ka2 ~] systemctl restart haproxy.service
效果展示
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
#注意,haproxy和lvs在同一虚拟路由上不能混合使用,完成此实验之前需要将lvs实验的配置还原,可以使用双主机模式将二者写在不同的虚拟路由上,使用不同的vip
当ka1关闭haproxy服务时,拥有vip的主机关闭了haproxy服务,导致客户机上链接被拒绝
[root@ka1 ~] systemctl stop haproxy.service
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 2211115 bytes 162252499 (154.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4776951 bytes 337045610 (321.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
curl: (7) Failed to connect to 172.25.254.100 port 80: Connection refused
可以通过killall命令查看haproxy的状态
[root@ka1 ~] killall -0 haproxy #查看已经关闭了haproxy服务的主机
haproxy: no process found #报错说明此时的haproxy的返回值为非0
[root@ka1 ~] echo $?
1
[root@ka2 ~] killall -0 haproxy #查看未关闭haproxy服务的主机
[root@ka2 ~] echo $?
0
4.编写脚本
[root@ka1 ~]# vim /etc/keepalived/test.sh
[root@ka1 ~]# cat /etc/keepalived/test.sh
#!/bin/bash
killall -0 haproxy
[root@ka2 ~]# vim /etc/keepalived/test.sh
[root@ka2 ~]# cat /etc/keepalived/test.sh
#!/bin/bash
killall -0 haproxy
5.使用vrrp script并编写检测脚本
vrrp_script check_haproxy {
script "/etc/keepalived/test.sh"
interval 1
weight -30
fall 2
rise 2
timeout 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
#preempt_delay 5s
# nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:1
}
unicast_src_ip 172.25.254.10
unicast_peer {
172.25.254.20
}
track_script {
check_haproxy
}
}
#ka2也需要配置,因为可能会涉及到抢占模式
[root@ka1 ~] systemctl restart keepalived.service
效果展示:
[root@ka1 ~] systemctl restart haproxy.service
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 2230513 bytes 163680073 (156.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4814430 bytes 339689220 (323.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
当ka1上关闭haproxy服务
[root@ka1 ~] systemctl stop haproxy.service
[root@ka1 ~] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.25.254.10 netmask 255.255.255.0 broadcast 172.25.254.255
inet6 fe80::20c:29ff:fec8:f6c4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:c8:f6:c4 txqueuelen 1000 (Ethernet)
RX packets 2254543 bytes 165457815 (157.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4862438 bytes 343051166 (327.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 28777 bytes 1496893 (1.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 28777 bytes 1496893 (1.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@servera ~] for i in {1..6}; do curl 172.25.254.100; done
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120
rs1 - 172.25.254.110
rs2 - 172.25.254.120