实验环境
VIP:192.168.239.98
VS1:192.168.239.128
VS2:192.168.239.130
RS1:192.168.239.131
RS2:192.168.239.132
操作系统CentOS。
VS1和VS2之间通过keepalived实现高可用的ipvs集群。后端RS1和RS2运行httpd服务
一、配置后端的两台Real Server
RS1:192.168.239.131
配置WEB服务
[root@centos6 ~]# yum install httpd
[root@centos6 ~]# vim /var/www/html/index.html
<h1>Real Server 131</h1>
配置VIP,绑定在lo网卡上,VIP的子网掩码设为255.255.255.255。同时为了避免地址冲突问题,得在各RS上修改内核参数,来限制arp响应和通告的级别,这里使用脚本的方式来实现。
[root@centos6 ~]# vim /data/lvs_dr.sh
#!/bin/bash
vip=192.168.239.98
mask='255.255.255.255'
dev=lo:1
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask
#route add -host $vip dev $dev
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
RS2:192.168.239.132
配置WEB服务
[root@centos7 ~]# yum install httpd
[root@centos7 ~]# vim /var/www/html/index.html
<h1>Real Server 132</h1>
配置VIP,绑定在lo网卡上,VIP的子网掩码设为255.255.255.255。同时为了避免地址冲突问题,得在各RS上修改内核参数,来限制arp响应和通告的级别,这里使用脚本的方式来实现。
[root@centos7 data]# vim lvs_dr.sh
#!/bin/bash
vip=192.168.239.98
mask='255.255.255.255'
dev=lo:1
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask
#route add -host $vip dev $dev
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
二、配置两台director(VS1和VS2)
1、安装keepalived
[root@centos7 ~]# yum install keepalived -y
2、编辑配置文件,配置VRRP实例和virtual_server模块
注意:这里的IPVS规则无需通过命令行方式配置,在配置文件中通过virtual_server模块来配置。因此ipvsadm命令用来查看ipvs规则即可
VS1的配置
[root@centos7 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node128
vrrp_mcast_group4 224.0.100.19
}
##VRRP实例
vrrp_instance VI_1 {
state MASTER //主节点
interface ens33
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass cafNzHqV
}
virtual_ipaddress {
192.168.239.98/24 dev ens33 label ens33:1 //这个地址用作VIP
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
##定义virtual_server,地址为192.168.239.98,该地址也就是VIP
virtual_server 192.168.239.98 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80 //定义sorry server是本机,因此本机也要配置http服务
real_server 192.168.239.131 80 { //后端的RS服务器
weight 1
HTTP_GET { //应用层检测
url {
path /
status_code 200
}
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.239.132 80 { //后端的RS服务器
weight 1
HTTP_GET { //应用层检测
url {
path /
status_code 200
}
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
}
}
}
VS2的配置
[root@centos7 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node130
vrrp_mcast_group4 224.0.100.19
}
vrrp_instance VI_1 {
state BACKUP //备用节点
interface ens33
virtual_router_id 51
priority 95
advert_int 2
authentication {
auth_type PASS
auth_pass cafNzHqV
}
virtual_ipaddress {
192.168.239.98/24 dev ens33 label ens33:1
}
notify_master "/etc/keepalived/notify.sh master" //调用通知脚本
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 192.168.239.98 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 192.168.239.131 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.239.132 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
}
}
}
上面两个VS的配置都调用了通知脚本,在VRRP实例中定义脚本的调用。
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
脚本的内容如下:
[root@centos7 keepalived]# cat notify.sh
#!/bin/bash
#
contact='root@localhost' ##定义接收用户,联系人
notify() { ##邮件通知函数
local mailsubject="$(hostname) to be $1, vip floating" ##邮件标题
local mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" ##邮件正文
echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
3、由于sorry server都定义是本机,因此两台提供keepalived高可用的VS服务器都要配置web服务。
[root@centos7 keepalived]# yum install httpd
[root@centos7 keepalived]# cat /var/www/html/index.html
FBI warning
三、测试
1、先启动备节点。
[root@centos7 keepalived]# systemctl start keepalived.service
[root@centos7 keepalived]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.239.130 netmask 255.255.255.0 broadcast 192.168.239.255
inet6 fe80::20c:29ff:fe65:a88f prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:65:a8:8f txqueuelen 1000 (Ethernet)
RX packets 86973 bytes 10307077 (9.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 93841 bytes 7775746 (7.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.239.98 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:65:a8:8f txqueuelen 1000 (Ethernet)
查看备节点的状态,这时发现先进入BACKUP状态,发现MASTER没有启动,在进入主节点状态。
[root@centos7 keepalived]# systemctl status keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since 日 2019-11-03 22:55:45 CST; 6s ago
Process: 19227 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 19228 (keepalived)
CGroup: /system.slice/keepalived.service
├─19228 /usr/sbin/keepalived -D
├─19229 /usr/sbin/keepalived -D
└─19230 /usr/sbin/keepalived -D
11月 03 22:55:45 centos7.cwj.com Keepalived_healthcheckers[19229]: Opening file '/etc/keepalived/keepalived.conf'.
11月 03 22:55:45 centos7.cwj.com Keepalived_healthcheckers[19229]: Activating healthchecker for service [192.168.239.98]:80
11月 03 22:55:45 centos7.cwj.com Keepalived_healthcheckers[19229]: Activating healthchecker for service [192.168.239.98]:80
11月 03 22:55:45 centos7.cwj.com Keepalived_vrrp[19230]: VRRP_Instance(VI_1) removing protocol VIPs.
11月 03 22:55:45 centos7.cwj.com Keepalived_vrrp[19230]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
11月 03 22:55:45 centos7.cwj.com Keepalived_vrrp[19230]: Using LinkWatch kernel netlink reflector...
11月 03 22:55:45 centos7.cwj.com Keepalived_vrrp[19230]: VRRP_Instance(VI_1) Entering BACKUP STATE
11月 03 22:55:45 centos7.cwj.com Keepalived_vrrp[19230]: Opening script file /etc/keepalived/notify.sh
11月 03 22:55:45 centos7.cwj.com Keepalived_vrrp[19230]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
11月 03 22:55:52 centos7.cwj.com Keepalived_vrrp[19230]: VRRP_Instance(VI_1) Transition to MASTER STATE
查看邮件,也可以发现是先进入BACKUO状态,在进入MASTER状态。
[root@centos7 keepalived]# mail
Heirloom Mail version 12.5 7/5/10. Type ? for help.
"/var/spool/mail/root": 2 messages 2 new
>N 1 root Sun Nov 3 22:55 18/681 "centos7.cwj.com to be backup, vip floating"
N 2 root Sun Nov 3 22:55 18/681 "centos7.cwj.com to be master, vip floating"
& 1
Message 1:
From root@centos7.cwj.com Sun Nov 3 22:55:45 2019
Return-Path: <root@centos7.cwj.com>
X-Original-To: root@localhost
Delivered-To: root@localhost.cwj.com
Date: Sun, 03 Nov 2019 22:55:45 +0800
To: root@localhost.cwj.com
Subject: centos7.cwj.com to be backup, vip floating
User-Agent: Heirloom mailx 12.5 7/5/10
Content-Type: text/plain; charset=us-ascii
From: root@centos7.cwj.com (root)
Status: R
2019-11-03 22:55:45: vrrp transition, centos7.cwj.com changed to be backup
& 2
Message 2:
From root@centos7.cwj.com Sun Nov 3 22:55:54 2019
Return-Path: <root@centos7.cwj.com>
X-Original-To: root@localhost
Delivered-To: root@localhost.cwj.com
Date: Sun, 03 Nov 2019 22:55:54 +0800
To: root@localhost.cwj.com
Subject: centos7.cwj.com to be master, vip floating
User-Agent: Heirloom mailx 12.5 7/5/10
Content-Type: text/plain; charset=us-ascii
From: root@centos7.cwj.com (root)
Status: R
2019-11-03 22:55:54: vrrp transition, centos7.cwj.com changed to be master
最后查看ipvs规则,发现规则以自动生成。
[root@centos7 keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.239.98:80 rr
-> 192.168.239.131:80 Route 1 0 0
-> 192.168.239.132:80 Route 1 0 0
客户端访问,定义的是轮询算法
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 131</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 131</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
2、这时启动主节点,发现地址会转移到主节点上,由主节点提供服务。
[root@centos7 keepalived]# systemctl start keepalived.service
[root@centos7 keepalived]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.239.128 netmask 255.255.255.0 broadcast 192.168.239.255
inet6 fe80::20c:29ff:fe60:4537 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:60:45:37 txqueuelen 1000 (Ethernet)
RX packets 79373 bytes 9900855 (9.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 98955 bytes 8530328 (8.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.239.98 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:60:45:37 txqueuelen 1000 (Ethernet)
查看ipvs规则
[root@centos7 keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.239.98:80 rr
-> 192.168.239.131:80 Route 1 0 0
-> 192.168.239.132:80 Route 1 0 0
同样主节点启动后也会收到邮件通知
[root@centos7 keepalived]# mail
Heirloom Mail version 12.5 7/5/10. Type ? for help.
"/var/spool/mail/root": 3 messages 1 new 2 unread
>N 1 root Sun Nov 3 23:06 18/683 "centos7.cwj.com to be master, vip floating"
此时客户端也能正常访问。
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 131</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 131</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
3、关闭后端的其中一台RS服务器
这时在查看ipvs规则发现只有一台主机工作,因此可以发现keepalived是可以对LVS进行健康状态检测的。
[root@centos7 keepalived]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.239.98:80 rr
-> 192.168.239.132:80 Route 1 0 0
这时客户端访问就只有一台主机提供服务了。
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>
[root@centos7 keepalived]# curl 192.168.239.98
<h1>Real Server 132</h1>