LVS 高可用性实现 -- ldirectord软件解决方案

一、lvs 高可用的必要性

LVS 不可用时 ：

Director不可用，整个系统将不可用；SPoF Single Point of Failure
解决方案：高可用，keepalived、heartbeat/corosync

RS 不可用时：

某RS不可用时，Director依然会调度请求至此RS
解决方案： 由Director对各RS健康状态进行检查，失败时禁用，成功时启用
常用解决方案：

• keepalived
• heartbeat/corosync
• ldirectord

ldirectord 检测方式：

• 网络层检测：icmp
• 传输层检测：端口探测
• 应用层检测，请求某关键资源

RS全不用时可以设置backup server, sorry serve

二、ldirectord软件

ldirectord是专门为LVS监控而编写的，用来监控lvs架构中服务器池（server pool） 的服务器状态。ldirectord 运行在 IPVS 节点上， ldirectord作为一个守护进程启动后会对服务器池中的每个真是服务器发送请求进行监控,如果 服务器没有响应 ldirectord 的请求，那么ldirectord 认为该服务器不可用，ldirectord 会运行 ipvsadm 对 IPVS表中该服务器进行删除，如果等下次再次检测有相应则通过
ipvsadm 进行添加：

ldirectord：监控和控制LVS守护进程，可管理LVS规则

包名：ldirectord-3.9.6-0rc1.1.1.x86_64.rpm

下载：[http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/]

三、 [LVS 高可用性实现 – ldirectord

1、lvs-dr环境部署

环境配置

role	host	ip	vip	os
Director-Server	node01	192.168.5.11	192.168.5.100	centos-7.8
Web1	node02	192.168.5.12	192.168.5.100	centos-7.8
Web1	node3	192.168.5.13	192.168.5.100	centos-7.8
Client	Client	Client	192.168.5.150	rhel-7.2

Director-Server 配置

[root@node01 ~]# yum install ipvsadm -y
[root@node01 ~]# vim /etc/init.d/lvs_dr
#!/bin/sh
# 
# Startup script handle the initialisation of LVS
# chkconfig: - 28 72
# description: Initialise the Linux Virtual Server for DR
# 
### BEGIN INIT INFO
# Provides: ipvsadm
# Required-Start: $local_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Short-Description: Initialise the Linux Virtual Server
# Description: The Linux Virtual Server is a highly scalable and highly
#   available server built on a cluster of real servers, with the load
#   balancer running on Linux.
# description: start LVS of DR
LOCK=/var/lock/ipvsadm.lock
VIP=192.168.5.100
RIP1=192.168.5.12
RIP2=192.168.5.13
DipName=ens33

. /etc/rc.d/init.d/functions
start() {
		PID=`ipvsadm -Ln | grep ${VIP} | wc -l`
		if  [ $PID -gt 0 ];
		then
		    echo "The LVS-DR Server is already running !"
		else
            #Set the Virtual IP Address
            /sbin/ifconfig ${DipName}:100 $VIP broadcast $VIP netmask 255.255.255.255 up
            /sbin/route add -host $VIP dev ${DipName}:100
            #Clear IPVS Table
            /sbin/ipvsadm -C
            #Set Lvs
            /sbin/ipvsadm -At $VIP:80 -s rr
            /sbin/ipvsadm -at $VIP:80 -r $RIP1:80 -g
            /sbin/ipvsadm -at $VIP:80 -r $RIP2:80 -g
            /sbin/ipvsadm -at $VIP:80 -r $RIP3:80 -g
            /bin/touch $LOCK
            #Run Lvs
            echo "starting LVS-DR Server is ok !"
        fi
} 
stop()      {

            #clear Lvs and vip
            /sbin/ipvsadm -C
            /sbin/route del -host $VIP dev ${DipName}:100
            /sbin/ifconfig ${DipName}:100 down >/dev/null
            rm -rf $LOCK
            echo "stopping LVS-DR server is ok !"
} 
status() {
       if [ -e $LOCK ];
       then
            echo "The LVS-DR Server is already running !"
       else
            echo "The LVS-DR Server is not running !"
       fi
} 
case "$1" in
  start)
       start
       ;;
  stop)
       stop
       ;;
  restart)
       stop
       start
       ;;
  status)
       status
       ;;
*)
       echo "Usage: $1 {start|stop|restart|status}"
       exit 1
esac
exit 0


[root@node01 ~]# chmod +x /etc/init.d/lvs_dr
[root@node01 ~]# /etc/init.d/lvs_dr start
Starting lvs_dr (via systemctl):                           [  OK  ]

Real Server配置 （node03配置雷同）

[root@node02 ~]# yum install httpd -y
[root@node02 ~]# echo "test web1 apge..." > /var/www/html/index.html
[root@node02 ~]# systemctl start httpd
[root@node02 ~]# curl localhost
test web1 apge...

[root@node02 ~]# vim  /etc/init.d/lvs_rs 
#!/bin/sh
# 
# Startup script handle the initialisation of LVS
# chkconfig: - 28 72
# description: Initialise the Linux Virtual Server for DR
# 
### BEGIN INIT INFO
# Provides: ipvsadm
# Required-Start: $local_fs $network $named
# Required-Stop: $local_fs $remote_fs $network
# Short-Description: Initialise the Linux Virtual Server
# Description: The Linux Virtual Server is a highly scalable and highly
#   available server built on a cluster of real servers, with the load
#   balancer running on Linux.
# description: start LVS of DR-RIP
LOCK=/var/lock/ipvsadm.lock
VIP=192.168.5.100
. /etc/rc.d/init.d/functions
start() {
	 PID=`ifconfig | grep lo:100 | wc -l`
	 if [ $PID -ne 0 ];
	 then
	     echo "The LVS-DR-RIP Server is already running !"
	 else
	    /sbin/ifconfig lo:100 $VIP netmask 255.255.255.255 broadcast $VIP up
	    /sbin/route add -host $VIP dev lo:100
	    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
	    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
	    echo "1" >/proc/sys/net/ipv4/conf/ens33/arp_ignore
	    echo "2" >/proc/sys/net/ipv4/conf/ens33/arp_announce
	    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
	    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
	    /bin/touch $LOCK
	    echo "starting LVS-DR-RIP server is ok !"
	 fi
	 } 
	 
stop() {
       /sbin/route del -host $VIP dev lo:100
       /sbin/ifconfig lo:100 down >/dev/null
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "0" >/proc/sys/net/ipv4/conf/ens33/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/ens33/arp_announce
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
       rm -rf $LOCK
       echo "stopping LVS-DR-RIP server is ok !"
}

status() {
     if [ -e $LOCK ];
     then
        echo "The LVS-DR-RIP Server is already running !"
     else
        echo "The LVS-DR-RIP Server is not running !"
     fi
} 

case "$1" in
   start)
        start
        ;;
   stop)
        stop
        ;;
   restart)
        stop
        start
        ;;
   status)
        status
        ;;
   *)
        echo "Usage: $1 {start|stop|restart|status}"
        exit 1
esac
exit 0


[root@node02 ~]# chmod +x /etc/init.d/lvs_rs
[root@node02 ~]# /etc/init.d/lvs_rs start
Reloading systemd:                                         [  OK  ]
Starting lvs_rs (via systemctl):                           [  OK  ]

测试：

2、故障出现

[root@node03 ~]# systemctl stop httpd
[root@node03 ~]# curl localhost
curl: (7) Failed connect to localhost:80; Connection refused

Client访问：

四、ldirectord 的应用

1、ldirectord的部署

[root@rhel7 ~]# ll ldirectord-3.9.6-0rc1.1.2.x86_64.rpm 
-rw-r--r--. 1 root root 100668 Feb 25  2021 ldirectord-3.9.6-0rc1.1.2.x86_64.rpm
[root@rhel7 ~]# yum install ldirectord-3.9.6-0rc1.1.2.x86_64.rpm -y
[root@node01 ~]# vim /etc/ha.d/ldirectord.cf 
checktimeout=3
checkinterval=1
#fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes
#logfile="/var/log/ldirectord.log"
#logfile="local0"
#emailalert="admin@x.y.z"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=no

# Sample for an http virtual service
virtual=192.168.5.100:80
	real=192.168.5.12:80 gate
	real=192.168.5.13:80 gate
	fallback=127.0.0.1:80 gate
	service=http
	scheduler=rr
	#persistent=600
	#netmask=255.255.255.255
	protocol=tcp
	checktype=negotiate
	checkport=80
	#request="index.html"
	#receive="Test Page"
	#virtualhost=www.x.y.z

[root@node01 ~]# systemctl restart ldirectord
[root@node01 ~]# systemctl is-active ldirectord
active
[root@node01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.5.100:80 rr
  -> 192.168.5.12:80              Route   1      0          0 

Client访问：

Client访问正常… …

2、故障恢复时

[root@node03 ~]# curl localhost
curl: (7) Failed connect to localhost:80; Connection refused
[root@node03 ~]# systemctl start httpd
[root@node03 ~]# curl localhost
test web2 apge...

Director-Server重新将故障恢复节点加入调度转发池

[root@node01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.5.100:80 rr
  -> 192.168.5.12:80              Route   1      0          0         
  -> 192.168.5.13:80              Route   1      0          0

Client访问：

Client访问正常… …