高可用集群KEEPALIVED

一.高可用集群

1.1 集群类型

• LB: 负载均衡
• HA: 高可用集群
• SPoF: 解决单点故障
• HPC: 高性能集群

1.2 系统可用性

A=MTBF/(MTBF+MTTR)

99.95%:(60*24*30)*(1-0.9995)=21.6分钟 #一般按一个月停机时间统计

1.3 实现高可用

提升系统高可用的解决方案：降低平均故障时间

解决方案：建立冗余机制

• active/passive 主/备
• active/active 双主
• active --> HEARTBEAT --> passive
• active HEARTBEAT active

二.Keepalived 部署

2.1 keepalived 简介

vrrp 协议的软件实现，原生设计目的为了高可用 ipvs服务

功能：

• 基于vrrp协议完成地址流动
• 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
• 为ipvs集群的各RS做健康状态检测
• 基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务

2.2 Keepalived 架构

• 用户空间核心组件：

        vrrp stack：VIP消息通告

        checkers：监测real server

        system call：实现 vrrp 协议状态转换时调用脚本的功能

        SMTP：邮件组件 IPVS

        wrapper：生成IPVS规则

        Netlink Reflector：网络接口

        WatchDog：监控进程

• 控制组件：提供keepalived.conf 的解析器，完成Keepalived配置
• IO复用器：针对网络目的而优化的自己的线程抽象
• 内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限

2.3 Keepalived 环境准备

2.3.1 ka1配置

[root@ka1 ~]# yum install keepalived.x86_64    -y
[root@ka1 ~]# systemctl start keepalived.service

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

2.3.2 ka2配置

[root@ka2 ~]# yum install keepalived.x86_64    -y
[root@ka2 ~]# systemctl start keepalived.service

2.3.3 测试

2.4 启用keepalived日志功能

[root@ka1 ~]# cat /etc/sysconfig/keepalived

KEEPALIVED_OPTIONS="-D -S 6"

[root@ka1 ~]# vim /etc/rsyslog.conf

[root@ka1 ~]# systemctl restart keepalived.service
[root@ka1 ~]# systemctl restart rsyslog.service

2.5 实现独立子配置文件

[root@ka1 ~]# mkdir /etc/keepalived/conf.d
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf

三.Keepalived 企业应用示例

3.1 抢占模式和非抢占模式

默认为抢占模式preempt，即当高优先级的主机恢复在线后，会抢占低先级的主机的master角色， 这样会使vip在KA主机中来回漂移，造成网络抖动， 建议设置为非抢占模式 nopreempt ，即高优先级主机恢复后，并不会抢占低优先级主机的master角色 非抢占模块下,如果原主机down机, VIP迁移至的新主机, 后续也发生down时,仍会将VIP迁移回原主机

注意：要关闭 VIP抢占，必须将各 keepalived 服务器state配置为BACKUP

ka1 主机配置

[root@ka1 ~]# vim /etc/keepalived/conf.d/route.conf
[root@ka1 ~]# cat  /etc/keepalived/conf.d/route.conf
vrrp_instance VI_1 {
        state BACKUP                     #必须将keepalived服务器设置为backup
        interface eth0
        virtual_router_id 100          
        priority 100                     #优先级高 
        nopreempt                        #非抢占模式
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass 1111
        }
        virtual_ipaddress {
                172.25.254.100/24 dev eth0 label eth0:1
        }
}

[root@ka1 ~]# systemctl restart keepalived.service

ka2配置

[root@ka2 ~]# vim /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
    state BACKUP                  #必须将keepalived服务器设置为backup
    interface eth0
    virtual_router_id 100
    priority 80                   #低优先级
    advert_int 1
    nopreempt                     #非抢占模式
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100/24 dev eth0 label eth0:1

    }
}

3.2.2 抢占延迟模式 preempt_delay

抢占延迟模式，即优先级高的主机恢复后，不会立即抢回VIP，而是延迟一段时间（默认300s）再抢回 VIP

 preempt_delay #     
#指定抢占延迟时间为#s，默认延迟300s

注意：需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict

ka1配置

ka2配置

测试

[root@ka1 ~]# watch -n 1 ifconfig

关闭ka1的keepalived后vip跳转到ka2

开启ka1后，vip不会立即跳回ka1，而是根据设置的延时抢占时间后抢占vip

3.3 VIP单播配置

默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以替换成单播，减少网络流量

注意：启用 vrrp_strict 时，不能启用单播

ka1配置

ka2配置

3.4 Keepalived 通知脚本配置

创建通知脚本

#!/bin/bash
mail_dest='1421304181@qq.com'

mail_send()
{
        mail_subj='$HOSTNAME to be $1 vip move'
        mail_msg='`date +%F\ %T`:vrrp move $HOSTNAME chage $1'
        echo $mail_msg | mail -s "mail_sub" $mail_dst
}

case $1 in
        master)
                send_message master
                ;;
        backup)
                send_message backup
                ;;
        fault)
                send_message fault
                ;;
        *)
                ;;
esac

给执行权限

[root@ka1 ~]# chmod +x  /etc/keepalived/mail.sh

[root@ka1 ~]# yum install sendmail   #下载邮件服务器

添加邮件配置

[root@ka1 ~]# cat /etc/mail.rc
 set from=1421304181@qq.com
 set smtp=smtp.qq.com
 set smtp-auth-user=1421304181@qq.com
 set smtp-auth-password=fmgqbgkvyunaifci
 set smtp-auth=login
 set ssl-verify=ignore

测试邮件

[root@ka1 ~]# echo tes | mail -s test 1421304181@qq.com

调用脚本：在 vrrp_instance VI_1 语句块的末尾加下面行

notify_master "/etc/keepalived/notify.sh master"
 notify_backup "/etc/keepalived/notify.sh backup"
 notify_fault "/etc/keepalived/notify.sh fault"

双主分别实现httpd和mysql服务的调度

在RS1和RS2中准备httpd和mysql环境

在RS1中：
[root@rs1 ~]# ip addr add 172.25.254.200/32 dev lo
 [root@rs1 ~]# yum install mariadb-server  -y
 root@rs1 ~]# vim /etc/my.cnf
 server-id=1
 [root@rs1 ~]# systemctl enable --now mariadb
 [root@rs1 ~]# mysql -e "grant ALL on *.* to lee@'%' identified by 'lee'"
 [root@rs1 ~]# mysql -ulee -plee -h172.25.254.101 -e 'select @@server_id'
 +-------------+
 | @@server_id |
 +-------------+
 |           
             1 |
 +-------------+

在RS2中：
[root@rs2 ~]# ip addr add 172.25.254.200/32 dev lo
 [root@rs2 ~]# yum install mariadb-server  -y
 [root@rs2 ~]# vim /etc/my.cnf
 server-id=2
 [root@rs2 ~]# systemctl enable --now mariadb
 [root@rs2 ~]# mysql -e "grant ALL on *.* to lee@'%' identified by 'lee'"
 [root@rs2 ~]# mysql -ulee -plee -h172.25.254.102 -e 'select @@server_id'
 +-------------+
 | @@server_id |
 +-------------+
 |           
             1 |
 +-------------+

ka1配置

#配置双主模式
[root@rhel7-ka1 ~]# vim /etc/keepalived/keepalived.conf
 vrrp_instance web {
 state MASTER
 interface ens33
 virtual_router_id 100
 priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100 dev ens33 label ens33:0
    }
 }
 vrrp_instance sql {
    state BACKUP
    interface ens33
    virtual_router_id 200
    priority 80
    advert_int 1
   authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.200 dev ens33 label ens33:1
    }
 }
 include "/etc/keepalived/conf.d/web.conf"
 include "/etc/keepalived/conf.d/sql.conf"
 [root@rhel7-ka1 ~]# vim /etc/keepalived/conf.d/web.conf
 virtual_server 172.25.254.100 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    protocol TCP
    sorry_server 172.25.254.30 80
    real_server 172.25.254.101 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 172.25.254.102 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
 }
 [root@rhel7-ka1 ~]# vim /etc/keepalived/conf.d/sql.conf
 virtual_server 172.25.254.200 3306 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP
    real_server 172.25.254.101 3306 {
        weight 1
        TCP_CHECK {
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 3306
        }
    }
    real_server 172.25.254.102 3306 {
        weight 1
        TCP_CHECK {
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 3306
        }
    }
 }

ka2配置

[root@ka1 keepalived]# vim /etc/keepalived/keepalived.conf
 vrrp_instance web {
    state BACKUP                #备机
    interface eth0
    virtual_router_id 50
    priority 80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.100 dev eth0 label eth0:1
    }
 }
 vrrp_instance sql {
    state MASTER            #主机
    interface eth0
    virtual_router_id 60
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.254.200 dev eth0 label eth0:1
    }
 }
 include "/etc/keepalived/conf.d/web.conf"
 include "/etc/keepalived/conf.d/sql.conf"
[root@ka1 keepalived]#  vim /etc/keepalived/conf.d/web.conf
 virtual_server 172.25.254.100 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    protocol TCP
    sorry_server 172.25.254.30 80
    real_server 172.25.254.10 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 172.25.254.102 80 {
        weight 1
        HTTP_GET {
            url {
                path /
                status_code 200
            }
            connect_timeout 1
            nb_get_retry 3
            delay_before_retry 1
        }
    }
 }
 [root@rhel7-ka2 ~]# vim /etc/keepalived/conf.d/web.conf
 virtual_server 172.25.254.200 3306 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP
    real_server 172.25.254.101 3306 {
        weight 1
        TCP_CHECK {
            connect_timeout 5
            nb_get_retry 3
            delay_before_retry 3
connect_port 3306
 }
 }
 real_server 172.25.254.102 3306 {
 weight 1
 TCP_CHECK {
 connect_timeout 5
 nb_get_retry 3
 delay_before_retry 3
 connect_port 3306
 }
 }
 }

使用ipvsadm查看

[root@ka1 keepalived]#  ipvsadm -Ln
 IP Virtual Server version 1.2.1 (size=4096)
 Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port               Forward Weight ActiveConn InActConn

TCP  172.25.254.100:80 wrr
    -> 172.25.254.100:80                  Route    1        0        0
    -> 172.25.254.101:80                  Route    1        0        0
TCP  172.25.254.200:3306 rr 
    -> 172.25.254.101:3306                Route    1        0        0
    -> 172.25.254.102:3306                Route    1        0        0

测试

[root@ka1 keepalived]#  mysql -ulee -plee -h172.25.254.200 -e 'select @@server_id'
 +-------------+
 | @@server_id |
 +-------------+
 |           
2 |
 +-------------+
[root@ka2 keepalived]#  mysql -ulee -plee -h172.25.254.200 -e 'select @@server_id'
 +-------------+
 | @@server_id |
 +-------------+
 |           
1 |
 +-------------+
[root@ka1 keepalived]#  mysql -ulee -plee -h172.25.254.200 -e 'select @@server_id'

利用脚本实现主从角色切换

[root@ka1 keepalived]# vim /mnt/check_lee.sh
 #!/bin/bash
 [ ！ -f "/mnt/lee" ]
[root@ka1 keepalived]#  chmod  +x /mnt/check_lee.sh
[root@ka1 keepalived]# vim /etc/keepalived/keepalived.conf

 vrrp_script check_lee {
 script "/mnt/check_lee.sh"
 interval 1
 weight -30
 fall 2
 rise 2
 timeout 2
 }
 vrrp_instance web {
 state MASTER
 interface ens33
 virtual_router_id 50
 priority 100
 advert_int 1
 }
 authentication {
 auth_type PASS
 auth_pass 1111
 }
 virtual_ipaddress {
 172.25.254.100 dev ens33 label ens33:0
 }
 track_script {
 check_lee
 }
[root@ka1 keepalived]# touch /mnt/lee
[root@ka1 keepalived]# tail -f /var/log/messages