最近给公司用heartbeat + ipvsadm + ldirectord做了一个web应用的集群,现在在这里做一下小结!
4台服务器:host1.example.com host2.example.com host3.example.com host4.example.com
操作系统均为CentOS 5.5 x86_64, ip 分配如下:
host1.example.com 168.168.2.161 eth0
host2.example.com 168.168.2.162 eth0
host3.example.com 168.168.2.163 eth0
host4.example.com 168.168.2.164 eth0
网关GW:168.168.2.1
对外的服务的虚拟IP:168.168.2.160
现在host1,host2用来做load balance。
1. 对host1做如下配置:
1)编辑/etc/hosts文件
vi /etc/hosts
168.168.2.160 host.example.com
168.168.2.161 host1.example.com host1
168.168.2.162 host1.example.com host2
168.168.2.163 host1.example.com host3
168.168.2.164 host1.example.com host4
2)安装ipvsadm
yum -y install ipvsadm
现在运行ipvsadm查看状态
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
3)添加heartbeat用户组和用户
getent group haclient >/dev/null || groupadd -r -g 694 haclient
getent passwd hacluster >/dev/null || useradd -r -g haclient -d /var/lib/heartbeat/cores/hacluster -s /sbin/nologin -c "cluster user" -u 694 hacluster
4)安装heartbeat
yum -y install heartbeat-devel heartbeat-gui heartbeat-stonith heartbeat-pils
yum -y install heartbeat
5)安装ldirectord
yum -y install heartbeat-ldirectord
查看heartbeat状态
rpm -qa | grep heartbeat
heartbeat-2.1.3-3.el5.centos
heartbeat-pils-2.1.3-3.el5.centos
heartbeat-ldirectord-2.1.3-3.el5.centos
heartbeat-stonith-2.1.3-3.el5.centos
heartbeat-devel-2.1.3-3.el5.centos
heartbeat-gui-2.1.3-3.el5.centos
6)拷贝相关配置文件到/etc/ha.d/目录
/cp /usr/share/doc/heartbeat-2.1.3/{ha.cf,authkeys,haresources} /etc/ha.d
/cp /usr/share/doc/heartbeat-ldirectord-2.1.3/ldirectord.cf /etc/ha.d
7)编辑配置文件
cd /etc/ha.d/
a)编辑 ha.cf
vi ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 5
deadtime 20
warntime 10
udpport 694 #通讯端口
ucast eth0 168.168.2.162 #(host2上面的ip改为168.168.2.161)
auto_failback on
node host1.example.com #此处节点名需要是对应主机的uname -n的输出结果
node host2.example.com
ping 168.168.2.1 #ip设置为该网段的网关
repawn hacluster /usr/lib64/heartbeat/ipfail #因为系统是64位的,这里是/usr/lib64/,如果是32位系统,就是/usr/lib/
apiauth ipfail gid=haclient uid=hacluster
use_logd no
b)编辑 authkeys
vi authkeys
auth 3
3 md5 Hello!
修改authkeys的权限
chmod 600 authkeys
注:authkeys的权限必须是600
c)编辑 haresources
vi haresoureces
host1.example.com IPaddr::168.168.2.160/24/eth0:0/168.168.2.255 ldirectord::ldirectord.cf lvs_lo
注1:主机名(uname -n) IPaddr::$VIP/24/虚拟的网卡设备/广播地址 脚本
注2:host1和host2上的此文件必须完全一致
d)编辑 ldirectord
vi ldirectord.cf
checktimeout = 10
checkinterval = 5
autoreload = yes
logfile = "/var/log/ldirectord.log"
emailalert = "sysadmin@example.com"
emailalertfreq = 3600
emailalertstatus = all
quiescent = yes
#注:这里选择yes时,需要修改内核配置
#echo 1 > /proc/sys/net/ipv4/vs/expire_quiescent_template
#echo 1 > /proc/sys/net/ipv4/vs/expire_nodest_conn
virtual=168.168.2.160:80
real=168.168.2.161:80 gate weight 8
real=168.168.2.162:80 gate weight 8
real=168.168.2.163:80 gate weight 10
real=168.168.2.164:80 gate weight 10
service=http
scheduler=wrr
protocol=tcp
checktype=connect
8)编辑/etc/sysctl.conf 文件
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.eth0.send_redirects = 0
net.ipv4.conf.eth1.send_redirects = 0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.vs.expire_quiescent_template = 1
net.ipv4.vs.expire_nodest_conn = 1
然后执行sysctl -p
9)测试ldirectord 是否配置成功
service ldirectord start
运行ipvsadm,有大致如下输出:
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP dict.cn:http wrr
-> host1.example.com:http Local 8 1 0
-> host2.example.com:http Route 8 0 0
-> host3.example.com:http Route 10 0 0
-> host4.example.com:http Route 10 0 0
停掉ldirectord,service ldirectord stop
2. 对host2做同host1相同的配置,标注的地方做一下修改就可以了
3. haresources 里面的lvs_lo脚本
vi lvs_lo
#!/bin/sh
VIP=168.168.2.160
GW=168.168.2.1
. /etc/rc.d/init.d/functions
case "$1" in
start)
echo "lo Virtual IP closing"
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
/sbin/route del $VIP
sysctl -p >/dev/null 2>&1
ifconfig lo:0 down
/sbin/ipvsadm --set 30 5 60
/sbin/arping -I eth0 -c 5 -s $VIP $GW >/dev/null 2>&1
;;
stop)
echo "lo Virtual IP starting"
/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
chmod +x lvs_lo
mv lvs_lo /etc/ha.d/resource.d/ #将lvs_lo脚本交给heartbeat来管理
4. 配置host1, host2, host3和host4
vi /etc/rc.local
/sbin/ifconfig lo:0 168.168.2.160 netmask 255.255.255.255 up
/sbin/route add -host 168.168.2.160 dev lo:0
vi /etc/sysctl.conf #注host1,host2中已经设置过了
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
保存后执行 sysctl -p
5. 启动host1 和 host2 上面的 heartbeat
/etc/init.d/heartbeat start
在host1上面运行ipvsadm时,有大致如下输出
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP dict.cn:http wrr
-> host1.example.com:http Local 8 1 0
-> host2.example.com:http Route 8 0 0
-> host3.example.com:http Route 10 0 0
-> host4.example.com:http Route 10 0 0
这时在host2上面运行ipvsadm,输出如下
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
把host1上面的heartbeat停掉,再在host1和host2上面运行ipvsadm的时候,发现host2已经将服务接管过去了
将heartbeat加入到系统启动项里面 chkconfig --levels 235 heartbeat on
6. 最后重启你的4台服务器。
到此为止,整个heartbeat + ldirectord来管理web负载均衡就完成了,整个集群当中,4台服务器都对外服务,可以根据实际情况,调整各台服务器的权重。
这个过程当中,可能在运行heartbeat的时候,检测到对方节点总是死掉,这时候你应该看看系统的防火墙!
运行 system-config-securitylevel ,在customize 里面的other port 添加 ha-cluster:udp ,保存,退出!