一. keepalived简介
Keepalived是HA cluster的一种实现方案,其以vrrp协议为实现基础,在N台具有相同功能的路由器或服务器中选出一个master或多个backup,这些路由器或服务器会共用一个对外提供服务的VIP。在此HA cluster集群中,master主机会发送组播来跟backup主机进行通信确认双方是否存活。当backup主机在指定的时间内没有收到vrrp组播报文则认为master主机宕机了,此时就会在所有的backup主机中根据vrrp协议来选举出新的master主机来保证服务的高可用性。
keepalived主要有三个模块,分别是ipvs wrapper、checkers和vrrp stack。ipvs wrapper模块为keepalived的核心,负责主进程的启动、维护以及全局配置文件的加载和解析。checkers负责健康检查,包括常见的各种检查方式。vrrp stack模块是来实现VRRP协议的。
二 keepalived的重要功能
keepalived 有三个重要的功能,分别是:
管理LVS负载均衡软件
实现LVS集群节点的健康检查
作为系统网络服务的高可用性(failover)
三 keepalived高可用故障转移的原理
Keepalived高可用五福之间故障转移,是通过VRRP来实现的。在Keepalived服务工作时,主Master节点会不断地向节点发送(多播的方式)心跳消息,用来告诉备Backup节点自己还活着。当主节点发生故障时,无法给备节点发送心跳消息,如果别节点无法继续监测来自主节点的心跳。就会调用自身的接管系统,接管主节点的ip资源和服务。当节点恢复的时候,备节点又会释放主节点故障时自身接管的ip资源和服务,恢复到原来的备用角色。
四 keepalived实现httpd负载均衡机高可用
环境说明centos8和centos8
主机名 | ip | 部署功能 |
---|---|---|
master | 192.168.143.101 | httpd和keepalived |
slave | 192.168.143.102 | httpd和keepalived |
keepalived安装 master 和 slave
//关闭防火墙与SELINUX
systemctl stop firewalld
systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
setenforce 0
sed -ri 's/^(SELINUX=).*/\1disabled/g' /etc/selinux/config
//安装基础包
dnf -y install epel-release vim wget
dnf -y -y install keepalived
//查看安装生成文件
rpm -ql keepalived
/etc/keepalived //配置文件
/etc/keepalived/keepalived.conf
/etc/sysconfig/keepalived //主配置文件
/usr/bin/genhash
/usr/lib/.build-id
/usr/lib/.build-id/0a
/usr/lib/.build-id/0a/410997e11c666114ca6d785e58ff0cc248744e
/usr/lib/.build-id/6f
/usr/lib/.build-id/6f/ba0d6bad6cb5ff7b074e703849ed93bebf4a0f
/usr/lib/systemd/system/keepalived.service //服务控制文件
安装httpd master 和 slave
dnf -y install httpd
systemctl enable --now httpd
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.
//web区别
[root@master ~]# echo web1 > /var/www/html/index.html
master 和 slave web页面
配置主keepalived
[root@master keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
router_id lb01
}
vrrp_instance VI_1 {
state MASTER
interface ens33 //这里是修改成本机的网卡名称
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.143.200
}
}
virtual_server 192.168.143.200 80 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.143.101 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.143.102 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
[root@master keepalived]# systemctl enable --now keepalived
Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.
配置备keepalived
[root@slave keepalived]# cat keepalived.conf
! Configuration File for keepalived
global_defs {
router_id lb02
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.143.200
}
}
virtual_server 192.168.143.200 80 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.143.101 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.143.102 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
[root@slave keepalived]# systemctl enable --now keepalived.service
Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.
查看VIP在哪里
[root@master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:39:f6:1c brd ff:ff:ff:ff:ff:ff
inet 192.168.143.101/24 brd 192.168.143.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.143.200/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe39:f61c/64 scope link
valid_lft forever preferred_lft forever
[root@slave keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:07:05:66 brd ff:ff:ff:ff:ff:ff
inet 192.168.143.102/24 brd 192.168.143.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe07:566/64 scope link
valid_lft forever preferred_lft forever
修改内核参数,开启监听VIP功能 master 和 slave
此步可做可不做,该功能可用于仅监听VIP的时候
echo 'net.ipv4.ip_nonlocal_bind = 1' >>/etc/sysctl.conf
sysctl -p
net.ipv4.ip_nonlocal_bind = 1
让keepalived监控httpd负载均衡机
//在master上编写脚本
[root@master]# mkdir /scripts
[root@master]# cd /scripts
[root@master scripts]# cat check_apache.sh
#!/bin/bash
httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l)
if [ $httpd_status -lt 1 ];then
systemctl stop keepalived
fi
[root@master scripts]# chmod +x check_apache.sh
[root@master scripts]# cat notify.sh
#! /bin/bash
#!/bin/bash
VIP=$2
sendmail (){
subject="${VIP}'s server keepalived state is translate"
content="`date +'%F %T'`: `hostname`'s state change to master"
echo $content | mail -s "$subject" hylzz2651@163.com
}
case "$1" in
master)
httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l)
if [ $httpd_status -lt 1 ];then
systemctl start httpd
fi
sendmail
;;
backup)
httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l)
if [ $httpd_status -gt 0 ];then
systemctl stop httpd
fi
;;
*)
echo "Usage:$0 master|backup VIP"
;;
esac
[root@master scripts]# chmod +x notify.sh
//在slave上编写脚本
[root@slave ~]# mkdir /scripts
[root@slave ~]# cd /scripts/
[root@slave scripts]# cat notify.sh
#!/bin/bash
VIP=$2
sendmail (){
subject="${VIP}'s server keepalived state is translate"
content="`date +'%F %T'`: `hostname`'s state change to master"
echo $content | mail -s "$subject" hylzz2651@163.com
}
case "$1" in
master)
httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l)
if [ $httpd_status -lt 1 ];then
systemctl start httpd
fi
sendmail
;;
backup)
httpd_status=$(ps -ef|grep -Ev "grep|$0"|grep '\bhttpd\b'|wc -l)
if [ $httpd_status -gt 0 ];then
systemctl stop httpd
fi
;;
*)
echo "Usage:$0 master|backup VIP"
;;
esac
[root@slave scripts]# chmod +x notify.sh
配置keepalived加入监控脚本的配置
//配置主keepalived
[root@master ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id lb01
}
vrrp_script httpd_check {
script "/scripts/check_apache.sh"
interval 10
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.143.200
}
track_script {
httpd_check
}
notify_master "/scripts/notify.sh master 192.168.143.200"
notify_backup "/scripts/notify.sh backup 192.168.143.200"
}
virtual_server 192.168.143.200 80 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.143.101 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.143.102 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
[root@master ~]# systemctl restart keepalived.service
//配置备keepalived
[root@slave ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id lb02
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.143.200
}
notify_master "/scripts/notify.sh master 192.168.143.200"
notify_backup "/scripts/notify.sh backup 192.168.143.200"
}
virtual_server 192.168.143.200 80 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.143.101 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.143.102 80 {
weight 1
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
[root@slave ~]# systemctl restart keepalived.service
注意应该访问192.168.143.200
当主服务器httpd服务停止或宕机vip访问自动到备用服务器
#master
//开启keepalived和httpd
[root@master ~]# systemctl start keepalived
[root@master ~]# systemctl start httpd
[root@master ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:80 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:80 [::]:*
LISTEN 0 128 [::]:22 [::]:*
#slave
//开启keepalived
[root@slave ~]# systemctl start keepalived
[root@slave ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
#master
//模拟master挂掉
[root@master ~]# systemctl stop httpd
[root@master ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
#slave
//此时的slave会因为master挂掉,自动上位,启动httpd
[root@slave ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:80 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:80 [::]:*
LISTEN 0 128 [::]:22 [::]:*
注意应该访问192.168.143.200
主服务器修复后,如何恢复VIP
#master
//先重启启动master的httpd,再启动keepalived
[root@master ~]# systemctl restart httpd
[root@master ~]# systemctl restart keepalived
[root@master ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:80 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:80 [::]:*
LISTEN 0 128 [::]:22 [::]:*
#slave
//此时slave会让位把httpd停止
[root@slave ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
注意应该访问192.168.143.200