快速了解高可用集群KEEPALIVED

LB：Load Balance 负载均衡
LVS/HAProxy/nginx（http/upstream, stream/upstream）
HA：High Availability 高可用集群
数据库、Redis
SPoF: Single Point of Failure，解决单点故障
HPC：High Performance Computing 高性能集群

1.2 系统可用性

SLA ： Service-Level Agreement 服务等级协议（提供服务的企业与客户之间就服务的品质、水准、性能等方面所达成的双方共同认可的协议或契约）

A = MTBF / (MTBF+MTTR ）

99.95%:(60*24*30)*(1-0.9995)=21.6 分钟 # 一般按一个月停机时间统计

指标： 99.9%, 99.99%, 99.999%,99.9999%

1.3 系统故障

硬件故障：设计缺陷、 wear out （损耗）、非人为不可抗拒因素

软件故障：设计缺陷 bug

1.4 实现高可用

提升系统高用性的解决方案：降低 MTTR- Mean Time To Repair( 平均故障时间 )

解决方案：建立冗余机制

active/passive 主/备
active/active 双主
active --> HEARTBEAT --> passive
active <--> HEARTBEAT <--> active

1.5.VRRP：Virtual Router Redundancy Protocol

虚拟路由冗余协议 , 解决静态网关单点风险

物理层:路由器、三层交换机
软件层:keepalived

1.5.1 VRRP 相关术语

虚拟路由器：Virtual Router
虚拟路由器标识：VRID(0-255)，唯一标识虚拟路由器
VIP：Virtual IP
VMAC：Virutal MAC (00-00-5e-00-01-VRID)
物理路由器：

master：主设备
backup：备用设备
priority：优先级

1.5.2 VRRP 相关技术

通告：心跳，优先级等；周期性

工作方式：抢占式，非抢占式

安全认证：

无认证
简单字符认证：预共享密钥
MD5

工作模式：

主/备：单虚拟路由器
主/主：主/备（虚拟路由器1），备/主（虚拟路由器2）

二.Keepalived 部署

2.1 keepalived 简介

vrrp 协议的软件实现，原生设计目的为了高可用 ipvs 服务

官网： http://keepalived.org/

功能：

基于vrrp协议完成地址流动
为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
为ipvs集群的各RS做健康状态检测
基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务

2.2 Keepalived 架构

官方文档：

https://keepalived.org/doc/

http://keepalived.org/documentation.html

用户空间核心组件：

                vrrp stack：VIP消息通告

                checkers：监测real server

                system call：实现 vrrp 协议状态转换时调用脚本的功能

                SMTP：邮件组件

                IPVS wrapper：生成IPVS规则

                Netlink Reflector：网络接口

                WatchDog：监控进程

控制组件：提供keepalived.conf 的解析器，完成Keepalived配置
IO复用器：针对网络目的而优化的自己的线程抽象
内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限

2.3 Keepalived 环境准备

各节点时间必须同步：ntp, chrony
关闭防火墙及SELinux
各节点之间可通过主机名互相通信：非必须
建议使用/etc/hosts文件实现：非必须
各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信：非必须

2.3.1 为了简化实验，在这里只使用四台虚拟机，不需要客户端主机（这里每个虚拟机都是RHEL7.9系统）。

主机名	ip	vip
ka1.com	192.168.10.10	192.168.10.100
ka2.com	192.168.10.20	192.168.10.100
realserver1.com	192.168.10.110	null
realserver2.com	192.168.10.120	null

2.3.2 实验第一步肯定是关闭每个虚拟机的防火墙和selinux

#这里给出一个虚拟机关闭防火墙和selinux

[root@realserver1 ~]# systemctl stop firewalld
[root@realserver1 ~]# setenforce 0

2.3.3 给realserver1.com和realserver2.com开启httpd服务和在/var/www/html/下写一个内容

[root@realserver1 ~]# yum install httpd -y #可能不需要下载，虚拟机自带

[root@realserver1 ~]# systemctl enable --now httpd.service

[root@realserver1 html]# echo 192.168.10.110 > /var/www/html/index.html

[root@realserver2 ~]# yum install httpd -y

[root@realserver2 ~]# systemctl enable --now httpd.service
[root@realserver2 html]# echo 192.168.10.120 > /var/www/html/index.html

2.3.4 在 ka1.com或ka2.com进行测试访问：

[root@ka1 ~]# curl 192.168.10.110
192.168.10.110
[root@ka1 ~]# curl 192.168.10.120
192.168.10.120

[root@ka2 ~]# curl 192.168.10.110
192.168.10.110
[root@ka2 ~]# curl 192.168.10.120
192.168.10.120

2.4 Keepalived 相关文件

软件包名：keepalived
主程序文件：/usr/sbin/keepalived
主配置文件：/etc/keepalived/keepalived.conf
配置文件示例：/usr/share/doc/keepalived/
Unit File：/lib/systemd/system/keepalived.service
Unit File的环境配置文件：/etc/sysconfig/keepalived

RHEL7 中可能会遇到一下 bug

systemctl restart keepalived # 新配置可能无法生效

systemctl stop keepalived;systemctl start keepalived # 无法停止进程，需要 kill 停止

2.5 Keepalived 安装

2.5.1 在ka1.com和ka2.com安装 keepalived

[root@ka1 ~]# yum install keepalived.x86_64 -y

[root@ka1 ~]# systemctl enable --now keepalived
[root@ka1 ~]# ps axf |grep keepalived
6629 pts/3 S+ 0:00 \_ grep --color=auto keepalived
4317 ? Ss 0:00 /usr/sbin/keepalived -D
4318 ? S 0:01 \_ /usr/sbin/keepalived -D
4319 ? S 0:01 \_ /usr/sbin/keepalived -D

[root@ka2 ~]# yum install keepalived.x86_64 -y

[root@ka2 ~]# systemctl enable --now keepalived
[root@ka2 ~]# ps axf |grep keepalived
5967 pts/2 S+ 0:00 \_ grep --color=auto keepalived
4170 ? Ss 0:00 /usr/sbin/keepalived -D
4171 ? S 0:01 \_ /usr/sbin/keepalived -D
4172 ? S 0:00 \_ /usr/sbin/keepalived -D

2.6 KeepAlived 配置说明

2.6.1 配置文件组成部分

配置文件： /etc/keepalived/keepalived.conf

配置文件组成

GLOBAL CONFIGURATION

        Global definitions：定义邮件配置，route_id，vrrp配置，多播地址等

VRRP CONFIGURATION

        VRRP instance(s)：定义每个vrrp虚拟路由器

LVS CONFIGURATION

        Virtual server group(s)

        Virtual server(s)： LVS集群的VS和RS

2.6.2 配置语法说明

#帮助

man keepalived.conf

2.6.2.1 全局配置

! Configuration File for keepalived

global_defs {

        notification_email {

          3036136367@qq.com         #keepalived 发生故障切换时邮件发送的目标邮箱，可以按行区

分写多个

                timiniglee-zln@163.com

        }

        notification_email_from keepalived@KA1.timinglee.org #发邮件的地址

        smtp_server 127.0.0.1 #邮件服务器地址

        smtp_connect_timeout 30 #邮件服务器连接 timeout

        router_id KA1.timinglee.org #每个 keepalived 主机唯一标识

                                                        #建议使用当前主机名，但多节点重名不影响

        vrrp_skip_check_adv_addr #对所有通告报文都检查，会比较消耗性能

                                                #启用此配置后，如果收到的通告报文和上一个报文是同一                                                 #个路由器，则跳过检查，默认值为全检查

        vrrp_strict                         #严格遵循vrrp协议

                                                #启用此项后以下状况将无法启动服务:

                                                #1.无VIP地址

                                                #2.配置了单播邻居

                                                #3.在VRRP版本2中有IPv6 地址

                                                #建议不加此项配置

        vrrp_garp_interval 0 #报文发送延迟，0表示不延迟

        vrrp_gna_interval 0 #消息发送延迟

        vrrp_mcast_group4 224.0.0.18 #指定组播 IP 地址范围：

        }

2.6.2.2 配置虚拟路由器

vrrp_instance VI_1 {

state MASTER

interface eth0 # 绑定为当前虚拟路由器使用的物理接口，如： eth0, 可以和 VIP 不在一

个网卡

virtual_router_id 51 # 每个虚拟路由器惟一标识 , 范围： 0-255 ，每个虚拟路由器此值必须唯一

                                #否则服务无法启动

                                #同属一个虚拟路由器的多个keepalived节点必须相同

                                #务必要确认在同一网络中此值必须唯一

priority 100                 #当前物理节点在此虚拟路由器的优先级，范围：1-254

                                    #值越大优先级越高,每个keepalived主机节点此值不同

advert_int 1                  #vrrp通告的时间间隔，默认1s

authentication {                 #认证机制

auth_type AH|PASS         #AH为 IPSEC 认证 ( 不推荐 ),PASS 为简单密码 ( 建议使用 )

uth_pass 1111                 #预共享密钥，仅前8 位有效

                                        #同一个虚拟路由器的多个keepalived节点必须一样

  }

virtual_ipaddress {         #虚拟 IP, 生产环境可能指定上百个 IP 地址

<IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL>

172.25.254.100                 #指定VIP ，不指定网卡，默认为 eth0, 注意：不指定 /prefix, 默认 32

172.25.254.101/24 dev eth1

172.25.254.102/24 dev eth2 label eth2:1

}

}

示例：

#配置ka1端

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf #红色是修改的内容

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.com
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}
[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

#配置ka2端

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA2.com
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100 #相同id管理同一个虚拟路由
priority 70 #低优先级
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}

[root@ka2 ~]# systemctl restart keepalived.service
[root@ka2 ~]#

#测试

[root@ka1 ~]# tcpdump -i eth0 -nn host 224.0.0.18

#在开启一个ka1会话

[root@ka1 ~]# systemctl stop keepalived.service

[root@ka1 ~]# systemctl restart keepalived.service

2.6.2.3 如何使用ping 192.168.10.100 使其ping通？

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.com
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
vrrp_iptables
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}

[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA2.com
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
vrrp_iptables
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}
[root@ka2 ~]# systemctl restart keepalived.service
[root@ka2 ~]#
#测试：

[root@ka1 ~]# ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.109 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.045 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.058 ms
--- 192.168.10.100 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.045/0.070/0.109/0.029 ms
或者

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id KA1.com
vrrp_skip_check_adv_addr
   #vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
#vrrp_iptables
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}

[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA2.com
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
#vrrp_iptables
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}
[root@ka2 ~]# systemctl restart keepalived.service
[root@ka2 ~]#

#测试：

[root@ka2 ~]# ping 192.168.10.100 -c2
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.256 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.717 ms

--- 192.168.10.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.256/0.486/0.717/0.231 ms

2.6.2.4 启用keepalived日志功能

示例：

#在这只展示ka2主机，还需在ka1主机上进行配置

[root@ka2 ~]# vim /etc/sysconfig/keepalived

KEEPALIVED_OPTIONS="-D -S 6"
[root@ka2 ~]# vim /etc/rsyslog.conf

local6.* /var/log/keepalived.log
[root@ka2 ~]# systemctl restart rsyslog.service
[root@ka2 ~]# systemctl restart keepalived.service
[root@ka2 ~]#
[root@ka2 ~]# tail /var/log/keepalived.log
Aug 12 19:25:03 ka2 Keepalived_healthcheckers[6742]: Remote SMTP server [192.168.200.1]:25 connected.
Aug 12 19:25:03 ka2 Keepalived_healthcheckers[6742]: Error reading data from remote SMTP server [192.168.200.1]:25.
Aug 12 19:25:05 ka2 Keepalived_healthcheckers[6742]: Error reading data from remote SMTP server [192.168.200.1]:25.
Aug 12 19:25:05 ka2 Keepalived_healthcheckers[6742]: Timeout connecting server [192.168.201.100]:443.
Aug 12 19:25:05 ka2 Keepalived_healthcheckers[6742]: Check on service [192.168.201.100]:443 failed after 3 retry.
Aug 12 19:25:05 ka2 Keepalived_healthcheckers[6742]: Removing service [192.168.201.100]:443 from VS [192.168.200.100]:443
Aug 12 19:25:05 ka2 Keepalived_healthcheckers[6742]: Lost quorum 1-0=1 > 0 for VS [192.168.200.100]:443
Aug 12 19:25:05 ka2 Keepalived_healthcheckers[6742]: Remote SMTP server [192.168.200.1]:25 connected.
Aug 12 19:25:06 ka2 Keepalived_healthcheckers[6742]: Error reading data from remote SMTP server [192.168.200.1]:25.
Aug 12 19:25:09 ka2 Keepalived_healthcheckers[6742]: Error reading data from remote SMTP server [192.168.200.1]:25.

2.6.2.5 实现独立子配置文件

当生产环境复杂时， /etc/keepalived/keepalived.conf 文件中内容过多，不易管理

将不同集群的配置，比如：不同集群的 VIP 配置放在独立的子配置文件中利用 include 指令可以实现包含

子配置文件

格式：

include /path/file

示例：

[root@ka1 ~]# mkdir -p /etc/keepalived/conf.d
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.com
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
#vrrp_iptables
}

include /etc/keepalived/conf.d/*.conf #配置子配置文件
#vrrp_instance VI_1 {
# state MASTER
# interface eth0
# virtual_router_id 100
# priority 100
# advert_int 1
# authentication {
# auth_type PASS
# auth_pass 1111
# }
# virtual_ipaddress {
#            192.168.10.100/24 dev eth0 label eth0:1
# }
#}
[root@ka1 ~]# vim /etc/keepalived/conf.d/192.168.10.100.conf

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
            192.168.10.100/24 dev eth0 label eth0:1
    }
}

#测试：

[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

[root@ka1 ~]# ifconfig

三.Keepalived 企业应用示例

3.1 实现master/slave的 Keepalived 单主架构

3.1.1 MASTER配置

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf #红色是修改的内容

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.com
vrrp_skip_check_adv_addr
vrrp_strict                #添加此选项无法访问vip，可以用nft list ruleset查看
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}
[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

3.1.2 BACKUP配置

# 配置文件和 master 基本一致，只需修改三行

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA2.com
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100 #相同id管理同一个虚拟路由
priority 70 #低优先级
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}

[root@ka2 ~]# systemctl restart keepalived.service
[root@ka2 ~]#

3.1.3 抓包观察

tcpdump -i eth0 -nn host 224.0.0.18

3.2 抢占模式和非抢占模式

3.2.1 非抢占模式 nopreempt

默认为抢占模式 preempt ，即当高优先级的主机恢复在线后，会抢占低先级的主机的 master 角色，

这样会使 vip 在 KA 主机中来回漂移，造成网络抖动，

建议设置为非抢占模式 nopreempt ，即高优先级主机恢复后，并不会抢占低优先级主机的 master 角色

非抢占模块下 , 如果原主机 down 机 , VIP 迁移至的新主机 , 后续也发生 down 时 , 仍会将 VIP 迁移回原主机

注意：要关闭 VIP抢占，必须将各 keepalived 服务器state配置为BACKUP

#ka1 主机配置

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 100         # 优先级高
advert_int 1         # 非抢占模式
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:1
}
}

#ka2 主机配置

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 70         # 优先级低
advert_int 1         # 非抢占模式
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:1
}
}

#测试

[root@ka1 ~]# ifconfig

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:11:87:ee txqueuelen 1000 (Ethernet)
[root@ka1 ~]# systemctl stop keepalived.service
[root@ka1 ~]# ifconfig

[root@ka1 ~]# systemctl start keepalived.service
[root@ka1 ~]# ifconfig

[root@ka2 ~]# ifconfig

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:be:89:23 txqueuelen 1000 (Ethernet)

3.2.2 抢占延迟模式 preempt_delay

抢占延迟模式，即优先级高的主机恢复后，不会立即抢回 VIP ，而是延迟一段时间（默认 300s ）再抢回VIP

preempt_delay # # 指定抢占延迟时间为 #s ，默认延迟 300s

注意：需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict

示例：

#ka1 主机配置

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 100         # 优先级高
advert_int 1
preempt_delay 5s        # 抢占延迟5 s
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:1
}
}

#ka2 主机配置

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 70         # 优先级低
advert_int 1
#nopreempt
preempt_delay 5s           # 抢占延迟5 s
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:1
}
}

#测试

[root@ka1 ~]# ifconfig

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:11:87:ee txqueuelen 1000 (Ethernet)

[root@ka1 ~]# systemctl stop keepalived.service
[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]# ifconfig #第一次测试没到5s

[root@ka1 ~]# ifconfig

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:11:87:ee txqueuelen 1000 (Ethernet)

3.3 VIP单播配置

默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以替换成单播，减少网络流量

注意：启用 vrrp_strict 时，不能启用单播

# 在所有节点 vrrp_instance 语句块中设置对方主机的 IP ，建议设置为专用于对应心跳线网络的地址，而非使

用业务网络

unicast_src_ip <IPADDR> # 指定发送单播的源 IP

unicast_peer {

<IPADDR> # 指定接收单播的对方目标主机 IP

......

}

# 启用 vrrp_strict 时，不能启用单播 , 否则服务无法启动 , 并在 messages 文件中记录下面信息

Jun 16 17:50:06 centos8 Keepalived_vrrp[23180]: (m44) Strict mode does not

support authentication. Ignoring.

Jun 16 17:50:06 centos8 Keepalived_vrrp[23180]: (m44) Unicast peers are not supported in strict mode

Jun 16 17:50:06 centos8 Keepalived_vrrp[23180]: Stopped - used 0.000606 user

time, 0.000000 system time

Jun 16 17:50:06 centos8 Keepalived[23179]: Keepalived_vrrp exited with permanent

error CONFIG. Terminating

Jun 16 17:50:06 centos8 systemd[1]: keepalived.service: Succeeded.

Jun 16 17:50:06 centos8 Keepalived[23179]: Stopped Keepalived v2.0.10

(11/12,2018)

示例：

#ka1 主机配置

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.com
vrrp_skip_check_adv_addr
#vrrp_strict                #注释此参数，与vip单播模式冲突
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
#vrrp_iptables
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
unicast_src_ip 192.168.10.10        #本机IP
unicast_peer {
   192.168.10.20                                #指向对方主机IP

                                                            #如果有多个keepalived,再加其它节点的IP

}
}

#在ka2主机中配置

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA2.com
vrrp_skip_check_adv_addr
#vrrp_strict                        #注释此参数，与vip单播模式冲突
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
#vrrp_iptables
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 70
advert_int 1
#nopreempt
#preempt_delay 5s
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
unicast_src_ip 192.168.10.20        #本机ip
unicast_peer {
192.168.10.10                                #对端主机IP
}
}

抓包查看单播效果

[root@ka1 ~]# tcpdump -i eth0 -nn src host 192.168.10.10 and dst 192.168.10.20
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:09:53.085663 IP 192.168.10.10 > 192.168.10.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
21:09:54.087269 IP 192.168.10.10 > 192.168.10.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
21:09:55.089031 IP 192.168.10.10 > 192.168.10.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
21:09:56.090698 IP 192.168.10.10 > 192.168.10.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
21:09:57.092075 IP 192.168.10.10 > 192.168.10.20: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
21:09:57.093591 ARP, Request who-has 192.168.10.20 tell 192.168.10.10, length 28
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

[root@ka1 ~]# systemctl stop keepalived.service
[root@ka1 ~]#

#查看ka2的效果

[root@ka2 ~]# tcpdump -i eth0 -nn src host 192.168.10.20 and dst 192.168.10.10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:10:41.155968 ARP, Reply 192.168.10.20 is-at 00:0c:29:be:89:23, length 28
21:10:54.180086 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:10:55.182935 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:10:56.184278 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:10:57.185480 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:10:58.186923 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:10:59.187974 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:10:59.188671 ARP, Request who-has 192.168.10.10 tell 192.168.10.20, length 28
21:11:00.188489 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:11:01.189416 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
21:11:02.190762 IP 192.168.10.20 > 192.168.10.10: VRRPv2, Advertisement, vrid 100, prio 70, authtype simple, intvl 1s, length 20
^C
11 packets captured
12 packets received by filter
0 packets dropped by kernel

3.4 Keepalived 通知脚本配置

当 keepalived 的状态变化时，可以自动触发脚本的执行，比如：发邮件通知用户

默认以用户 keepalived_script 身份执行脚本

如果此用户不存在，以 root 执行脚本可以用下面指令指定脚本执行用户的身份

global_defs {

......

script_user <USER>

......

}

3.4.1 通知脚本类型

当前节点成为主节点时触发的脚本

notify_master <STRING>|<QUOTED-STRING>

当前节点转为备节点时触发的脚本

notify_backup <STRING>|<QUOTED-STRING>

当前节点转为 “ 失败 ” 状态时触发的脚本

notify_fault <STRING>|<QUOTED-STRING>

通用格式的通知触发机制，一个脚本可完成以上三种状态的转换时的通知

notify <STRING>|<QUOTED-STRING>

当停止 VRRP 时触发的脚本

notify_stop <STRING>|<QUOTED-STRING>

3.4.2 脚本的调用方法

在 vrrp_instance VI_1 语句块的末尾加下面行

notify_master "/etc/keepalived/notify.sh master"

notify_backup "/etc/keepalived/notify.sh backup"

notify_fault "/etc/keepalived/notify.sh fault"

3.4.3 创建通知脚本

[root@ka1 ~]# vim /etc/keepalived/mail.sh

#!/bin/bash
mail_dst="3036136367@qq.com"
send_message()
{
mail_sub="$HOSTNAME to be $1 vip move"
mail_msg="`date +%F\ %T`: vrrp move $HOSTNAME chage $1"
echo $mail_msg | mail -s "$mail_sub" $mail_dst
}

case $1 in
master)
send_message master
;;
backup)
send_message backup
;;
fault)
send_message fault
;;
*)
;;
esac

3.4.4 邮件配置

安装邮件发送工具

[root@ka1 ~]# yum install mailx -y

QQ 邮箱配置

[root@ka1 ~]# vim /etc/mail.rc

set from=3036136367@qq.com
set smtp=smtp.qq.com
set smtp-auth-user=3036136367@qq.com
set smtp-auth-password=tjjgllbktkvsdebe
set smtp-auth=login
set ssl-verify=ignore

发送测试邮件

[root@ka1 ~]# echo hello |mail -s test 3036136367@qq.com

[root@ka2 ~]# echo 12345 | mail -s xixi 3036136367@qq.com

3.4.5 实战案例：实现 Keepalived 状态切换的通知脚本

#在ka1和ka2进行配置，因为配置一样，所以我只展示其中一个

[root@ka1 ~]# vim /etc/keepalived/mail.sh

#!/bin/bash
mail_dst="3036136367@qq.com"
send_message()
{
mail_sub="$HOSTNAME to be $1 vip move"
mail_msg="`date +%F\ %T`: vrrp move $HOSTNAME chage $1"
echo $mail_msg | mail -s "$mail_sub" $mail_dst
}

case $1 in
master)
send_message master
;;
backup)
send_message backup
;;
fault)
send_message fault
;;
*)
;;
esac

[root@ka1 ~]# chmod +x /etc/keepalived/mail.sh
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:1
}
unicast_src_ip 192.168.10.10
unicast_peer {
192.168.10.20
}
notify_master "/etc/keepalived/mail.sh master"
notify_backup "/etc/keepalived/mail.sh backup"
notify_fault "/etc/keepalived/mail.sh fault"
}

[root@ka1 ~]# ifconfig

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:11:87:ee txqueuelen 1000 (Ethernet)

#模拟master故障
[root@ka1 ~]# systemctl restart keepalived.service

[root@ka1 ~]# systemctl stop keepalived.service

3.5 实现 master/master 的 Keepalived 双主架构

master/slave 的单主架构，同一时间只有一个 Keepalived 对外提供服务，此主机繁忙，而另一台主机却

很空闲，利用率低下，可以使用 master/master 的双主架构，解决此问题。

master/master 的双主架构：

即将两个或以上 VIP 分别运行在不同的 keepalived 服务器，以实现服务器并行提供 web 访问的目的，提高服务器资源利用率

#ka1主机配置

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
state MASTER #主
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}

vrrp_instance VI_2 {
state BACKUP #备
interface eth0
virtual_router_id 200
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.200/24 dev eth0 label eth0:2
}
}
[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

#ka2 主机配置

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}

vrrp_instance web {
state MASTER
interface eth0
virtual_router_id 200
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.200/24 dev eth0 label eth0:2
}
}
[root@ka2 ~]# systemctl restart keepalived.service
[root@ka2 ~]#

测试：

[root@ka1 ~]# ifconfig

eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.100 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:11:87:ee txqueuelen 1000 (Ethernet)

[root@ka2 ~]# ifconfig

eth0:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.10.200 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:be:89:23 txqueuelen 1000 (Ethernet)

实战案例：三个节点的三主架构实现

#每个虚拟机每个节点都只有一个 MASTER，剩下的都是BACKUP（当然四个或更多节点都是只有一个MASTER）。

# 第一个节点 ka1 配置：

Vrrp instance 1 ： MASTER ，优先级 100

Vrrp instance 2 ： BACKUP ，优先级 80

Vrrp instance 3 ： BACKUP ，优先级 60

# 第二个节点 ka2 配置：

Vrrp instance 1 ： BACKUP ，优先级 60

Vrrp instance 2 ： MASTER ，优先级 100

Vrrp instance 3 ： BACKUP ，优先级 80

# 第三个节点 ka3 配置：

Vrrp instance 1 ： BACKUP ，优先级 80

Vrrp instance 2 ： BACKUP ，优先级 60

Vrrp instance 3 ： MASTER ，优先级 100

3.6 实现IPVS的高可用性

3.6.1 IPVS相关配置

3.6.1.1 虚拟服务器配置结构

virtual_server IP port {

...

real_server {

...

}

real_server {

...

}

…

}

3.6.1.2 virtual server （虚拟服务器）的定义格式

virtual_server IP port         #定义虚拟主机 IP 地址及其端口

virtual_server fwmark int         #ipvs的防火墙打标，实现基于防火墙的负载均衡集群

virtual_server group string         #使用虚拟服务器组

3.6.1.3 虚拟服务器配置

virtual_server IP port {         #VIP和 PORT

delay_loop <INT>         #检查后端服务器的时间间隔

lb_algo rr|wrr|lc|wlc|lblc|sh|dh         #定义调度方法

lb_kind NAT|DR|TUN         #集群的类型 , 注意要大写

persistence_timeout <INT>         #持久连接时长

protocol TCP|UDP|SCTP         #指定服务协议 , 一般为 TCP

sorry_server <IPADDR> <PORT>         #所有 RS 故障时，备用服务器地址

real_server <IPADDR> <PORT> {         #RS的 IP 和 PORT

weight <INT>                                         #RS权重

notify_up <STRING>|<QUOTED-STRING>         #RS上线通知脚本

notify_down <STRING>|<QUOTED-STRING>         #RS下线通知脚本

HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK { ... }         #定义当前主机健康状态检测方法

        }

}

#注意:括号必须分行写,两个括号写在同一行,如: }} 会出错

3.6.1.4 应用层监测

应用层检测： HTTP_GET|SSL_GET

HTTP_GET|SSL_GET {

url {

path <URL_PATH>         #定义要监控的 URL

status_code <INT>         #判断上述检测机制为健康状态的响应码，一般为 200

}

connect_timeout <INTEGER>         #客户端请求的超时时长 , 相当于 haproxy 的 timeout server

nb_get_retry <INT>         #重试次数

delay_before_retry <INT>         #重试之前的延迟时长

connect_ip <IP ADDRESS>         #向当前 RS 哪个 IP 地址发起健康状态检测请求

connect_port <PORT>         #向当前 RS 的哪个 PORT 发起健康状态检测请求

bindto <IP ADDRESS>         #向当前 RS 发出健康状态检测请求时使用的源地址

bind_port <PORT>         #向当前 RS 发出健康状态检测请求时使用的源端口

}

3.6.1.5 TCP监测

传输层检测： TCP_CHECK

TCP_CHECK {

connect_ip <IP ADDRESS>         #向当前 RS 的哪个 IP 地址发起健康状态检测请求

connect_port <PORT>         #向当前 RS 的哪个 PORT 发起健康状态检测请求

bindto <IP ADDRESS>         #发出健康状态检测请求时使用的源地址

bind_port <PORT>         #发出健康状态检测请求时使用的源端口

connect_timeout <INTEGER>         #客户端请求的超时时长

                                                        #等于haproxy的timeout server

}

3.6.2 实战案例

3.6.2.1 实战案例1：实现单主的 LVS-DR 模式

准备 web 服务器并使用脚本绑定 VIP 至 web 服务器 lo 网卡

# 准备两台后端 RS 主机配置

[root@realserver1 ~]# yum install httpd -y #下载apache

[root@realserver1 html]# echo 192.168.10.110 > /var/www/html/index.html

[root@realserver1 ~]# ip a a 192.168.10.100/32 dev lo #临时的vip，当关机时这个需要重新配置

[root@realserver1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

[root@realserver1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore

[root@realserver1 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

[root@realserver1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

#永久配置vip

[root@realserver1 ~]# vim /etc/sysconfig/network-scripts/ifcfg-lo
DEVICE=lo
IPADDR0=127.0.0.1
NETMASK0=255.0.0.0
IPADDR1=192.168.10.100
NETMASK1=255.255.255.255
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback

#也可以在文档中写

[root@realserver1 all]# vim /etc/sysctl.d/arp.conf
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2

[root@realserver2 ~]# yum install httpd -y #下载apache

[root@realserver2 html]# echo 192.168.10.120 > /var/www/html/index.html

[root@realserver2 ~]# ip a a 192.168.10.100/32 dev lo
[root@realserver2 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

[root@realserver2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore

[root@realserver2 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

[root@realserver2 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

配置keepalived

#ka1 节点的配置

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

virtual_server 192.168.10.100 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
#persistence_timeout 50
protocol TCP

real_server 192.168.10.110 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}

real_server 192.168.10.120 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}
}

[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

#ka2节点的配置

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

virtual_server 192.168.10.100 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
#persistence_timeout 50
protocol TCP

real_server 192.168.10.110 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}

real_server 192.168.10.120 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}
}

访问测试结果

[root@ka2 ~]# for i in {1..5}
> do
> curl 192.168.10.100
> done
192.168.10.110
192.168.10.120
192.168.10.110
192.168.10.120
192.168.10.110

[root@ka1 ~]# yum install ipvsadm -y
[root@ka2 ~]# yum install ipvsadm -y

[root@ka1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 wrr
-> 192.168.10.110:80 Route 1 0 0
-> 192.168.10.120:80 Route 1 0 0
TCP 10.10.10.2:1358 rr persistent 50
-> 192.168.200.200:1358 Masq 1 0 0
TCP 10.10.10.3:1358 rr persistent 50

[root@ka2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 wrr
-> 192.168.10.110:80 Route 1 0 3
-> 192.168.10.120:80 Route 1 0 2
TCP 10.10.10.2:1358 rr persistent 50
-> 192.168.200.200:1358 Masq 1 0 0
TCP 10.10.10.3:1358 rr persistent 50

模拟故障

# 第一台 RS1 故障，自动切换至 RS2

[root@realserver1 ~]# systemctl stop httpd #当RS1故障
[root@realserver1 ~]#

[root@ka2 ~]# for i in {1..5}; do curl 192.168.10.100; done

# 全部流浪被定向到 RS2 中
192.168.10.120
192.168.10.120
192.168.10.120
192.168.10.120
192.168.10.120

[root@ka1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 wrr
-> 192.168.10.120:80 Route 1 0 0            #RS1 被踢出保留
TCP 10.10.10.2:1358 rr persistent 50
-> 192.168.200.200:1358 Masq 1 0 0
TCP 10.10.10.3:1358 rr persistent 50

#ka1 故障，自动切换至 ka2

[root@ka1 ~]# systemctl stop keepalived.service
[root@ka1 ~]#

[root@ka2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 wrr
-> 192.168.10.110:80 Route 1 0 1
-> 192.168.10.120:80 Route 1 0 1
TCP 10.10.10.2:1358 rr persistent 50
-> 192.168.200.200:1358 Masq 1 0 0
TCP 10.10.10.3:1358 rr persistent 50

3.6.2.2 实战案例2：实现双主的 LVS-DR 模式

[root@ka1 ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
   3036136367@qq.com
   timiniglee-zln@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.com        #另一个节点为KA2.com
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_mcast_group4 224.0.0.18
#vrrp_iptables
}
vrrp_script check_haproxy {
script "/etc/keepalived/scripts/haproxy.sh"
interval 1
weight -30
fall 2
rise 2
timeout 2
}

vrrp_instance VI_1 {
state MASTER        #在另一个结点上为BACKUP
interface eth0
virtual_router_id 100 #这个router_id必须相同
priority 100        #在另一个结点上为70
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1        #指定VIP
}
#track_script {
#   check_haproxy
# }
# unicast_src_ip 192.168.10.10
# unicast_peer {
#   192.168.10.20
# }
#notify_master "/etc/keepalived/mail.sh master"
#notify_backup "/etc/keepalived/mail.sh backup"
#notify_fault "/etc/keepalived/mail.sh fault"
}

vrrp_instance VI_2 {
state BACKUP        #在另一个结点上为MASTER
interface eth0
virtual_router_id 200
priority 70        #在另一个结点上为100
advert_int 1
#preempt_delay 5s
#nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.200/24 dev eth0 label eth0:2        #指定VIP2
}
# unicast_src_ip 192.168.10.10
# unicast_peer {
#   192.168.10.20
# }
}

#include "/etc/keepalived/conf.d/*.conf"
virtual_server 192.168.10.100 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
#persistence_timeout 50
protocol TCP

real_server 192.168.10.110 80 {        #指定RS1地址
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}

real_server 192.168.10.120 80 {        #指定RS2地址
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}
}

virtual_server 10.10.10.2 1358 {         #指定VIP2
delay_loop 6
lb_algo rr
lb_kind NAT
persistence_timeout 50
protocol TCP

sorry_server 192.168.200.200 1358

real_server 192.168.200.2 1358 {        #指定RS3地址
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl3/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}

real_server 192.168.200.3 1358 {        #指定RS4地址
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334c
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334c
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}

virtual_server 10.10.10.3 1358 {        #指定VIP2
delay_loop 3
lb_algo rr
lb_kind NAT
persistence_timeout 50
protocol TCP

real_server 192.168.200.4 1358 {   #指定RS5地址
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl3/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}

real_server 192.168.200.5 1358 {        #指定RS6地址
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl3/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}

企业示例 : 双主分别实现 httpd 和 mysql 服务的调度

#准备实验素材

#在RS1中配置：

[root@realserver1 ~]# ip a a 192.168.10.200/32 dev lo

[root@realserver1 ~]# yum install mariadb-server -y
[root@realserver1 ~]# vim /etc/my.cnf #红色是编写的内容
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
[root@realserver1 ~]# systemctl enable --now mariadb.service
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@realserver1 ~]#[root@realserver1 ~]# mysql -e "grant ALL on *.* to lee@'%' identified by 'lee'"
[root@ka1 ~]# mysql -ulee -plee -h192.168.10.110 -e 'select @@server_id'
+-------------+
| @@server_id |
+-------------+
| 1 |
+-------------+

#在RS2中配置：

[root@realserver2 ~]# ip a a 192.168.10.200/32 dev lo
[root@realserver2 ~]# yum install mariadb-server -y

[root@realserver2 ~]# vim /etc/my.cnf
[mysqld]
server-id=2
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
[root@realserver2 ~]# vim /etc/my.cnf
[root@realserver2 ~]# systemctl enable --now mariadb.service
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.

[root@realserver2 ~]# mysql -e "grant ALL on *.* to lee@'%' identified by 'lee'"
[root@ka1 ~]# mysql -ulee -plee -h192.168.10.120 -e 'select @@server_id'
+-------------+
| @@server_id |
+-------------+
| 2 |
+-------------+

#配置双主模式

在ka1中配置：

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
state MASTER #主机
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}

vrrp_instance VI_2 {
state BACKUP #备机
interface eth0
virtual_router_id 200
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.200/24 dev eth0 label eth0:2
}
}

include "/etc/keepalived/conf.d/*.conf"

[root@ka1 ~]# vim /etc/keepalived/conf.d/web.conf
server 192.168.10.100 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
#persistence_timeout 50
protocol TCP

real_server 192.168.10.110 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}

real_server 192.168.10.120 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}
}

[root@ka1 conf.d]# vim sql.conf

server 192.168.10.200 3306 {
delay_loop 6
lb_algo rr
lb_kind DR
#persistence_timeout 50
protocol TCP

real_server 192.168.10.110 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
   connect_port 3306
}
}

real_server 192.168.10.120 3306 {
weight 1
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
   connect_port 3306
}
}
}

[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

#在ka2中配置：

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.100/24 dev eth0 label eth0:1
}
}
vrrp_instance web {
state MASTER
interface eth0
virtual_router_id 200
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
   192.168.10.200/24 dev eth0 label eth0:2
}
}
include "/etc/keepalived/conf.d/*.conf"

[root@ka2 ~]# mkdir -p /etc/keepalived/conf.d
[root@ka2 ~]# vim /etc/keepalived/conf.d/web.conf

server 192.168.10.100 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
#persistence_timeout 50
protocol TCP

real_server 192.168.10.110 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}

real_server 192.168.10.120 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
}
}
}

[root@ka2 ~]# vim /etc/keepalived/conf.d/sql.conf

server 192.168.10.200 3306 {
delay_loop 6
lb_algo rr
lb_kind DR
#persistence_timeout 50
protocol TCP

real_server 192.168.10.110 3306 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
   connect_port 3306
}
}

real_server 192.168.10.120 3306 {
weight 1
connect_timeout 3
nb_get_retry 2
delay_before_retry 2
   connect_port 3306
}
}
}

[root@ka2 ~]# systemctl restart keepalived.service
[root@ka2 ~]#

#注意：如果写在外部不能实现效果，那就在里面写（[root@ka2 ~]# vim /etc/keepalived/keepalived.conf），这样还知道自己哪里写错了

[root@ka2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.10.100:80 wrr
-> 192.168.10.110:80 Route 1 0 0
-> 192.168.10.120:80 Route 1 0 0
TCP 192.168.10.200:3306 rr
-> 192.168.10.110:3306 Route 1 0 0
-> 192.168.10.120:3306 Route 1 0 0
TCP 10.10.10.2:1358 rr persistent 50
-> 192.168.200.200:1358 Masq 1 0 0
TCP 10.10.10.3:1358 rr persistent 50

#测试

[root@ka1 conf.d]# mysql -ulee -plee -h192.168.10.200 -e 'select @@server_id'
+-------------+
| @@server_id |
+-------------+
| 1 |
+-------------+
[root@ka1 conf.d]# mysql -ulee -plee -h192.168.10.200 -e 'select @@server_id'
+-------------+
| @@server_id |
+-------------+
| 2 |
+-------------+

3.7 实现其它应用的高可用性 VRRP Script

keepalived 利用 VRRP Script 技术，可以调用外部的辅助脚本进行资源监控，并根据监控的结果实现优先

动态调整，从而实现其它应用的高可用性功能

参考配置文件： /usr/share/doc/keepalived/keepalived.conf.vrrp.localcheck

3.7.1 VRRP Script 配置

分两步实现：

定义脚本

vrrp_script：自定义资源监控脚本， vrrp 实例根据脚本返回值，公共定义，可被多个实例调用，定

义在 vrrp 实例之外的独立配置块，一般放在 global_defs 设置块之后。

通常此脚本用于监控指定应用的状态。一旦发现应用的状态异常，则触发对MASTER 节点的权重减至低于SLAVE 节点，从而实现 VIP 切换到 SLAVE 节点

vrrp_script <SCRIPT_NAME> {

script <STRING>|<QUOTED-STRING> # 此脚本返回值为非 0 时，会触发下面 OPTIONS 执行

OPTIONS

}

调用脚本

track_script ：调用 vrrp_script 定义的脚本去监控资源，定义在 VRRP 实例之内，调用事先定义的

vrrp_script

track_script {

SCRIPT_NAME_1

SCRIPT_NAME_2

}

3.7.1.1 定义 VRRP script

vrrp_script <SCRIPT_NAME> {         #定义一个检测脚本，在 global_defs 之外配置

script <STRING>|<QUOTED-STRING>         #shell命令或脚本路径

interval <INTEGER>         #间隔时间，单位为秒，默认 1 秒

timeout <INTEGER>         #超时时间

weight <INTEGER:-254..254>         #默认为 0, 如果设置此值为负数，

                                                        #当上面脚本返回值为非0时

                                                        #会将此值与本节点权重相加可以降低本节点权重，

# 即表示 fall.

                                                                #如果是正数，当脚本返回值为0，

                                                                #会将此值与本节点权重相加可以提高本节点权重

                                                                #即表示 rise.通常使用负值

fall <INTEGER>         #执行脚本连续几次都失败 , 则转换为失败，建议设为 2 以上

rise <INTEGER>         #执行脚本连续几次都成功，把服务器从失败标记为成功

user USERNAME [GROUPNAME]         #执行监测脚本的用户或组

init_fail         #设置默认标记为失败状态，监测成功之后再转换为成功状态

}

3.7.1.2 调用 VRRP script

vrrp_instance test {

... ...

track_script {

check_down

}

}

3.7.2 实战案例：利用脚本实现主从角色切换

[root@ka1 ~]# vim /mnt/check_lee.sh
#!/bin/bash
[ ! -f "/mnt/lee" ]

[root@ka1 ~]# chmod +x /mnt/check_lee.sh

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

vrrp_script check_lee {
script "/mnt/check_lee.sh"
interval 1
weight -30
fall 2
rise 2
timeout 2
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:1
}
track_script {
check_lee
}
}

[root@ka1 ~]# touch /mnt/lee
[root@ka1 ~]# tail -f /var/log/messages
Aug 17 15:52:43 ka1 Keepalived_vrrp[5333]: /mnt/check_lee.sh exited with status 1
Aug 17 15:52:44 ka1 Keepalived_vrrp[5333]: /mnt/check_lee.sh exited with status 1
Aug 17 15:52:45 ka1 Keepalived_vrrp[5333]: /mnt/check_lee.sh exited with status 1
Aug 17 15:52:46 ka1 Keepalived_vrrp[5333]: /mnt/check_lee.sh exited with status 1
Aug 17 15:52:47 ka1 Keepalived_vrrp[5333]: /mnt/check_lee.sh exited with status 1

3.7.3 实战案例：实现HAProxy高可用

# 在两个 ka1 和 ka2 先实现 haproxy 的配置，这里只展示ka1配置

[root@ka1 ~]# yum install haproxy -y

[root@ka1 ~]# vim /etc/haproxy/haproxy.cfg

listen webserver
bind *:80
server web1 192.168.10.110:80 check
server web2 192.168.10.120:80 check

# 在两个 ka1 和 ka2两个节点启用内核参数，这里只展示ka1配置

[root@ka1 ~]# vim /etc/sysctl.conf

net.ipv4.ip_nonlocal_bind=1

[root@ka1 ~]# sysctl -p
net.ipv4.ip_nonlocal_bind = 1

# 在 ka1 中编写检测脚本
[root@ka1 ~]# vim /etc/keepalived/scripts/haproxy.sh
#!/bin/bash
/usr/bin/killall -0 haproxy

# 在 ka1 中配置 keepalived

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

vrrp_script check_haproxy {
script "/etc/keepalived/scripts/haproxy.sh"
interval 1
weight -30
fall 2
rise 2
timeout 2
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:1
}
track_script {
check_haproxy
}
}

[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]#

#测试

[root@ka1 ~]# systemctl stop haproxy
[root@ka1 ~]# tail -f /var/log/messages
Aug 17 16:04:29 ka1 Keepalived_vrrp[7184]: /etc/keepalived/scripts/haproxy.sh exited with status 1
Aug 17 16:04:30 ka1 Keepalived_vrrp[7184]: /etc/keepalived/scripts/haproxy.sh exited with status 1
Aug 17 16:04:31 ka1 Keepalived_vrrp[7184]: /etc/keepalived/scripts/haproxy.sh exited with status 1
Aug 17 16:04:32 ka1 Keepalived_vrrp[7184]: /etc/keepalived/scripts/haproxy.sh exited with status 1