heartbeat + ipvsadm + ldirectord 实现CentOS5.5上的web负载均衡

最近给公司用heartbeat + ipvsadm + ldirectord做了一个web应用的集群，现在在这里做一下小结！

 

4台服务器：host1.example.com  host2.example.com host3.example.com host4.example.com

操作系统均为CentOS 5.5 x86_64, ip 分配如下：

host1.example.com 168.168.2.161   eth0

host2.example.com 168.168.2.162   eth0

host3.example.com 168.168.2.163   eth0 

host4.example.com 168.168.2.164   eth0 

网关GW：168.168.2.1

对外的服务的虚拟IP：168.168.2.160

 

现在host1，host2用来做load balance。

1. 对host1做如下配置：

1）编辑/etc/hosts文件

    vi /etc/hosts

    168.168.2.160   host.example.com

    168.168.2.161   host1.example.com  host1

    168.168.2.162   host1.example.com  host2

    168.168.2.163   host1.example.com  host3

    168.168.2.164   host1.example.com  host4

 

2）安装ipvsadm

     yum -y install ipvsadm

 

     现在运行ipvsadm查看状态

     IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
         -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

 

3）添加heartbeat用户组和用户

    getent group haclient >/dev/null || groupadd -r -g 694 haclient
    getent passwd hacluster >/dev/null || useradd -r -g haclient -d /var/lib/heartbeat/cores/hacluster -s /sbin/nologin -c "cluster user" -u 694 hacluster

 

4）安装heartbeat

     yum -y install heartbeat-devel heartbeat-gui heartbeat-stonith  heartbeat-pils

     yum -y install heartbeat

 

5）安装ldirectord

     yum -y install heartbeat-ldirectord

 

     查看heartbeat状态

     rpm -qa | grep heartbeat

     heartbeat-2.1.3-3.el5.centos
     heartbeat-pils-2.1.3-3.el5.centos
     heartbeat-ldirectord-2.1.3-3.el5.centos
     heartbeat-stonith-2.1.3-3.el5.centos
     heartbeat-devel-2.1.3-3.el5.centos
     heartbeat-gui-2.1.3-3.el5.centos

 

 6）拷贝相关配置文件到/etc/ha.d/目录

     /cp /usr/share/doc/heartbeat-2.1.3/{ha.cf,authkeys,haresources} /etc/ha.d
     /cp /usr/share/doc/heartbeat-ldirectord-2.1.3/ldirectord.cf /etc/ha.d

 

 7）编辑配置文件

      cd /etc/ha.d/

   a）编辑 ha.cf

      vi  ha.cf

      debugfile   /var/log/ha-debug

      logfile  /var/log/ha-log

      logfacility   local0

      keepalive   5

      deadtime   20

      warntime   10

      udpport     694   #通讯端口

      ucast  eth0  168.168.2.162    #(host2上面的ip改为168.168.2.161)

      auto_failback on

      node host1.example.com   #此处节点名需要是对应主机的uname -n的输出结果

      node host2.example.com

      ping 168.168.2.1       #ip设置为该网段的网关

      repawn hacluster /usr/lib64/heartbeat/ipfail   #因为系统是64位的，这里是/usr/lib64/，如果是32位系统，就是/usr/lib/

      apiauth  ipfail  gid=haclient  uid=hacluster

      use_logd no

 

   b）编辑 authkeys

        vi authkeys

        auth 3

        3 md5 Hello!

        修改authkeys的权限

        chmod 600 authkeys

        注：authkeys的权限必须是600

 

   c）编辑 haresources

        vi haresoureces

        host1.example.com IPaddr::168.168.2.160/24/eth0:0/168.168.2.255 ldirectord::ldirectord.cf lvs_lo

        注1：主机名(uname -n)  IPaddr::$VIP/24/虚拟的网卡设备/广播地址  脚本

        注2：host1和host2上的此文件必须完全一致

 

   d）编辑 ldirectord

        vi ldirectord.cf

        checktimeout = 10

        checkinterval = 5

        autoreload  = yes

        logfile = "/var/log/ldirectord.log"

        emailalert = "sysadmin@example.com"

        emailalertfreq =  3600

        emailalertstatus = all

        quiescent = yes

             #注：这里选择yes时，需要修改内核配置

             #echo 1 > /proc/sys/net/ipv4/vs/expire_quiescent_template

             #echo 1 > /proc/sys/net/ipv4/vs/expire_nodest_conn

        virtual=168.168.2.160:80

                     real=168.168.2.161:80 gate weight 8

                     real=168.168.2.162:80 gate weight 8

                     real=168.168.2.163:80 gate weight 10

                     real=168.168.2.164:80 gate weight 10

                     service=http

                     scheduler=wrr

                     protocol=tcp

                     checktype=connect

 

   8）编辑/etc/sysctl.conf 文件

        vi /etc/sysctl.conf

        net.ipv4.ip_forward = 1
        net.ipv4.conf.all.send_redirects = 0
        net.ipv4.conf.default.send_redirects = 0
        net.ipv4.conf.eth0.send_redirects = 0
        net.ipv4.conf.eth1.send_redirects = 0
        net.ipv4.conf.all.arp_ignore = 1
        net.ipv4.conf.all.arp_announce = 2
        net.ipv4.conf.lo.arp_ignore = 1
        net.ipv4.conf.lo.arp_announce = 2

        net.ipv4.vs.expire_quiescent_template = 1 
        net.ipv4.vs.expire_nodest_conn = 1
        然后执行sysctl -p

 

    9）测试ldirectord 是否配置成功

         service ldirectord start

         运行ipvsadm，有大致如下输出：

        IP Virtual Server version 1.2.1 (size=4096)
        Prot LocalAddress:Port Scheduler Flags
          -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
        TCP  dict.cn:http wrr
          -> host1.example.com:http       Local    8           1                 0
          -> host2.example.com:http       Route   8           0                 0
          -> host3.example.com:http       Route   10         0                 0
          -> host4.example.com:http       Route   10         0                 0

         停掉ldirectord，service ldirectord stop

 

2. 对host2做同host1相同的配置，标注的地方做一下修改就可以了

 

3. haresources 里面的lvs_lo脚本

    vi lvs_lo

        #!/bin/sh

        VIP=168.168.2.160
        GW=168.168.2.1

        . /etc/rc.d/init.d/functions

        case "$1" in
          start)
            echo "lo Virtual IP closing"
            echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
            echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
            echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
            echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
            /sbin/route del $VIP
            sysctl -p >/dev/null 2>&1
            ifconfig lo:0 down
            /sbin/ipvsadm --set 30 5 60
            /sbin/arping -I eth0 -c 5 -s $VIP $GW >/dev/null 2>&1
            ;;
          stop)
            echo "lo Virtual IP starting"
            /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up
            /sbin/route add -host $VIP dev lo:0
            echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
            echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
            echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
            echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
            sysctl -p >/dev/null 2>&1
            ;;
          *)
            echo "Usage: $0 {start|stop}"
            exit 1

        esac

        exit 0

 

        chmod +x lvs_lo

        mv lvs_lo /etc/ha.d/resource.d/ #将lvs_lo脚本交给heartbeat来管理

 

4. 配置host1, host2, host3和host4

    vi /etc/rc.local

    /sbin/ifconfig lo:0 168.168.2.160 netmask 255.255.255.255 up
    /sbin/route add -host 168.168.2.160 dev lo:0

 

    vi /etc/sysctl.conf    #注host1,host2中已经设置过了

    net.ipv4.conf.all.arp_ignore = 1
    net.ipv4.conf.all.arp_announce = 2
    net.ipv4.conf.lo.arp_ignore = 1
    net.ipv4.conf.lo.arp_announce = 2

    保存后执行 sysctl -p

 

5. 启动host1 和 host2 上面的 heartbeat

        /etc/init.d/heartbeat start

        在host1上面运行ipvsadm时，有大致如下输出

        IP Virtual Server version 1.2.1 (size=4096)
        Prot LocalAddress:Port Scheduler Flags
          -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
        TCP  dict.cn:http wrr
          -> host1.example.com:http       Local    8           1                 0
          -> host2.example.com:http       Route   8           0                 0
          -> host3.example.com:http       Route   10         0                 0
          -> host4.example.com:http       Route   10         0                 0

        这时在host2上面运行ipvsadm，输出如下

 

        IP Virtual Server version 1.2.1 (size=4096)
        Prot LocalAddress:Port Scheduler Flags
          -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

        把host1上面的heartbeat停掉，再在host1和host2上面运行ipvsadm的时候，发现host2已经将服务接管过去了

 

        将heartbeat加入到系统启动项里面   chkconfig --levels 235 heartbeat on

 

6. 最后重启你的4台服务器。

      到此为止，整个heartbeat + ldirectord来管理web负载均衡就完成了，整个集群当中，4台服务器都对外服务，可以根据实际情况，调整各台服务器的权重。

 

 

这个过程当中，可能在运行heartbeat的时候，检测到对方节点总是死掉，这时候你应该看看系统的防火墙！

运行 system-config-securitylevel ，在customize 里面的other port 添加 ha-cluster:udp ,保存，退出！