keepalived + rsync +inotify-tools实现双机热备

## 1.需求

实现数据交换系统的主备自动切换功能，当数据交换的master主机故障,网络故障以及数据交换mysql服务出现故障时，能自动切换到备机运行。

## 2.实现思路

采用keepalived+rsync+ inotify双backup模式，并设置不抢占资源。当master挂机，切换为backup,完成修复后，不会抢占为maste!避免不必要的切换。 假设两台主机如下。
•    master host：192.168.1.151，debain
•    backup host：192.168.1.152，debain

安装步骤如下：
1.    安装apt-get install keepalived
2.    安装apt-get install rsync
3.    安装apt-get install inotify-tools
4.    启动

## 3.安装keeplived

```
 apt-get install keeplived
```

#### 修改master主机上的/etc/keepalived/keepalived.conf配置文件

```
 cat /etc/keepalived/keepalived.conf
```


```
! Configuration File for keepalived

global_defs {
   router_id xchange_router     # all node must same
}

vrrp_instance xchange {
    state BACKUP                   # master or backup
    interface eth1                 # 热备通信网口
    virtual_router_id 55           # all node must same
    priority  100                  # 优先级，高的先运行。主机异常时切换到低优先级热备机，副机一般设为99
    advert_int 1                   # send keepalived msg every 1 second
    nopreempt                      # must need
    authentication {               # all node must same
        auth_type PASS
        auth_pass 1111
    }

    track_interface { #需要检测的其它
        eth0          #在些添加的网络接口会被检测，eth0不正常会切换热备服务
    }

    notify_master /etc/keepalived/to_master.sh  #切换为master时执行
    notify_backup /etc/keepalived/to_backup.sh  #切换为backup时执行
}
virtual_server 127.0.0.1 3306 { #表示需要检测的服务
    delay_loop 2
    lb_algo rr
    lb_kind NAT
    persistence_timeout 60
    protocol TCP
    real_server 127.0.0.1 3306 {
    weight 3
    notify_down /etc/keepalived/shutdown.sh #服务检测失败执行操作
    TCP_CHECK {
    connect_timeout 10
    nb_get_retry 3
    delay_before_retry 3
    connect_port 3306
    }
}
```

#### 修改backup主机上的/etc/keepalived/keepalived.conf

```
! Configuration File for keepalived

global_defs {
   router_id xchange_router     # all node must same
}

vrrp_instance xchange {
    state BACKUP                   # master or backup
    interface eth1                 # 热备通信网口
    virtual_router_id 55           # all node must same
    priority  99                   # 优先级，高的先运行。主机异常时切换到低优先级热备机，副机一般设为99
    advert_int 1                   # send keepalived msg every 1 second
    #nopreempt                     # must need
    authentication {               # all node must same
        auth_type PASS
        auth_pass 1111
    }

    track_interface { #需要检测的其它
        eth0          #在些添加的网络接口会被检测，eth0不正常会切换热备服务
    }

    notify_master /etc/keepalived/to_master.sh  #切换为master时执行
    notify_backup /etc/keepalived/to_backup.sh  #切换为backup时执行
}
virtual_server 127.0.0.1 3306 { #表示需要检测的服务
    delay_loop 2
    lb_algo rr
    lb_kind NAT
    persistence_timeout 60
    protocol TCP
    real_server 127.0.0.1 3306 {
    weight 3
    notify_down /etc/keepalived/shutdown.sh #服务检测失败执行操作
    TCP_CHECK {
    connect_timeout 10
    nb_get_retry 3
    delay_before_retry 3
    connect_port 3306
    }
}

```

#### 创建shell脚本，放在/etc/keepalived目录下

> 脚本1：shutdown.sh

```

#!/bin/bash
echo "shutdown.sh" >> /var/log/messages
killall keepalived
/etc/init.d/manager stop

```

>脚本2：to_master.sh

```

#!/bin/bash
echo "to_master" >> /var/log/messages
/etc/init.d/manager stop
/etc/init.d/manager start

```

>脚本3:to_backup.sh

```
#!/bin/bash

echo "to_backup" >> /var/log/messages
/etc/init.d/manager stop


```

>脚本4:keepalived_sync.sh,

```
#!/bin/bash

# usage: sh keepalived_sync.sh
 
USER=appbackup            # the account to login rsyc server. Notice, it's not the linux user
IP=192.168.1.152           # the rsync server's ip.主备对应修改，192.168.1.151/192.168.1.152
DES="xchange"
SRC="/usr/app/xchange/repository"
PASSWD_FILE="/etc/rsync_client.secret"
LOGFILE="/var/log/messages.log"

/usr/bin/inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f%e' -e close_write,modify,delete,create,attrib,move ${src} |  while read file
do
/usr/bin/rsync -vzrtopg --delete --progress --password-file=$PASSWD_FILE $SRC $USER@$IP::$DES
echo "${files} was rsynced" >>$LOGFILE 2>&1
done


```

#### keepalived的相关配置文件存储位置如下：

```
root@deb:/etc/keepalived$ tree /etc/keepalived/
/etc/keepalived/
├── keepalived.conf
├── to_backup.sh
├── shutdown.sh
├── to_master.sh
└── keepalived_sync.sh

1 directory, 5 files

```

## 安装Rsync

```
apt-get install rsync
```

#### 修改/etc/default/rsync文件

```
RSYNC_ENABLE=true
```

#### 配置rsyncd的/etc/rsyncd.conf

```
 cat /etc/rsyncd.conf
```

```
# sample rsyncd.conf configuration file

# GLOBAL OPTIONS

#motd file=/etc/motd
log file=/var/log/messages
# for pid file, do not use /var/run/rsync.pid if
# you are going to run rsync out of the init.d script.
# The init.d script does its own pid file handling,
# so omit the "pid file" line completely in that case.
# pid file=/var/run/rsyncd.pid
syslog facility=daemon
#socket options=

# MODULE OPTIONS

[xchange]

comment = public archive
path = /usr/app/xchange/repository
use chroot = yes
#    max connections=10
lock file = /var/lock/rsyncd
# the default for read only is yes...
read only = no
list = yes
uid = root
gid = root
#    exclude =
#    exclude from =
#    include =
#    include from =
auth users =appbackup
secrets file = /etc/rsyncd.secrets
strict modes = yes
#    hosts allow =
#    hosts deny =
ignore errors = no
ignore nonreadable = yes
transfer logging = no
#    log format = %t: host %h (%a) %o %f (%l bytes). Total %b bytes.
timeout = 600
refuse options = checksum dry-run
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

```

#### 修改/etc/rsync.secret

```
sudo cat /etc/rsync.secret
```

```
appbackup:passwd

```

> 修改/etc/rsync.secret文件属性:

```
 chown root:root /etc/rsync.secret
 chmod 600 /etc/rsync.secret
```

#### 启动rsync服务器

```
 service rsync restart
```

####  配置rsync客户端,

> 文件/etc/rsync_client.secret内容如下

```
passwd
```

> 修改文件权限
```
sudo chmod 600 /etc/rsync_client.secret
```

** 注意：如果不修改权限，则会报错ERROR: password file must not be other-accessible**

## 安装 inotify-tools

```
apt-get install inotify-tools
```

## 5.启动

#### 1. 启动master主机上的rsync
```
 service rsync restart
```
#### 2. 启动master主机上的keepalived
```
 service keepalived restart
```

#### 5. 查看启动日志
```
tail -f /var/log/messages.log
```

## 6.自动切换测试

#### 确认keepalived、rsync已经启动

```
ps -ef | grep keepalived
ps -ef | grep rsync
```
## 7.加入开机脚本
#echo "/data/sh/inotifyrsync.sh &"  >> /etc/rc.local

## 8.manager去掉开机启动

#### 切换测试

> 1.停止master主机上的mysql进程，查看master主机上的manager服务是否停止，检查backup主机是否启动manager服务。
> 2.或者关机master，查backup主机是否启动manager服务。


执行流程说明：
默认情况下主副机器都不会运行manager服务，主副机启动keepalived服务，首行执行优先级高的主机to_master脚本，启动manager服务,如果出现mysql停止或宕机情况，会执行shutdown.sh停止热备和manager服务，此时主副切换，副机执行to_master脚本,过程中如果主机修复，主机会抢占热备服务，副机执行to_backup脚本并停止manager服务，主机重新执行to_master并开启manager