MHA高可用搭建简述（基于日志点）

本文在主从的基础搭建，配置好本地yum源，

本文所依据配置结构

172.16.101.1 mha-server//现主机
172.16.101.3 mha-bak//备用主机
172.16.101.2 mha-slave//从机&监控机

安装 MHA 及基本环境配置

1、在所有的节点上安装 MHA node 所需的 perl 模块(DBD-mysql)

yum install perl-DBD-MySQL -y ---前面已经安装过了

2、在所有的节点上安装 mha 的 node 数据节点

yum localinstall mha4mysql-node-0.54-0.el6.noarch.rpm

3、安装完成后，看/usr/bin 下是否生成以下脚本文件：

[root@root mha4mysql-node-0.56]# cd /usr/bin
[root@root bin]# ll
total 44
-r-xr-xr-x. 1 root root 16367 Dec 30 00:43
apply_diff_relay_logs
-r-xr-xr-x. 1 root root 4807 Dec 30 00:43
filter_mysqlbinlog
-r-xr-xr-x. 1 root root 8261 Dec 30 00:43 purge_relay_logs
-r-xr-xr-x. 1 root root 7525 Dec 30 00:43 save_binary_logs
[root@root bin]#

4、在 monitor 节点（mha-slave）安装 MHA manager

yum localinstall mha4mysql-manager-0.55-0.el6.noarch.rpm

安装成功后，生成以下脚本文件：

[root@root bin]# cd /usr/bin
[root@root bin]# ls |grep "masterha_*"
masterha_check_repl
masterha_check_ssh
masterha_check_status
masterha_conf_host
masterha_manager
masterha_master_monitor
masterha_master_switch
masterha_secondary_check
masterha_stop
[root@root bin]#

5、配置主机名字（三台机器都要配置）

[root@localhost mysql]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.101.1 mha-server//现主机
172.16.101.3 mha-bak//备用主机
172.16.101.2 mha-slave//从机&MHA监控机

6、配置 SSH 登录无密码验证

每台都要配置到另外的两台免密登录（只演示其中一台）

[root@localhost bin]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
7a:7b:01:fe:84:ff:02:66:2a:07:34:4e:26:d1:7c:84 root@localhost.localdomain
The key's randomart image is:
+--[ RSA 2048]----+
|   o o.          |
|  . E .          |
|   . .           |
|  . =   .        |
|   * . .So       |
|    o  .* o      |
|     ..+.= .     |
|    . o. .=      |
|     o  .. o.    |
+-----------------+
[root@localhost bin]# cd ~/.ssh
[root@localhost .ssh]# cat id_rsa.pub >> authorized_keys
[root@localhost .ssh]# ll
total 12
-rw-r--r--. 1 root root  408 Sep 25 22:55 authorized_keys
-rw-------. 1 root root 1671 Sep 25 22:54 id_rsa
-rw-r--r--. 1 root root  408 Sep 25 22:54 id_rsa.pub
[root@localhost .ssh]# chmod 700 /root/.ssh
[root@localhost .ssh]# chmod 600 authorized_keys

————————————————————————————————————————————————————————————————————————————
[root@localhost .ssh]# ssh-copy-id -i id_rsa.pub root@172.16.101.1
The authenticity of host '172.16.101.1 (172.16.101.1)' can't be established.
RSA key fingerprint is 90:08:94:cc:66:cb:bd:b5:f0:05:41:e5:76:40:df:e1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.101.1' (RSA) to the list of known hosts.
root@172.16.101.1's password: 
Now try logging into the machine, with "ssh 'root@172.16.101.1'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.
————————————————————————————————————————————————————————————————————————————
[root@localhost .ssh]# ssh-copy-id -i id_rsa.pub root@172.16.101.3
The authenticity of host '172.16.101.3 (172.16.101.3)' can't be established.
RSA key fingerprint is 57:7a:eb:e5:e6:bf:ca:ba:17:20:70:a9:c7:d2:52:fa.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.101.3' (RSA) to the list of known hosts.
root@172.16.101.3's password: 
Now try logging into the machine, with "ssh 'root@172.16.101.3'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

配置 MHA

在 monitor（mha-slave）上执

1、创建 MHA 的工作目录

[root@root ~]# mkdir -p /etc/masterha

2、修改相关配置文件/etc/masterha/app1.cnf，内容如下：

小注：
对于配置文件内容详解，请参考官方文档：
https://code.google.com/p/mysql-master-ha/wiki/Parameters
MHA 主页地址及下载等：
https://code.google.com/p/mysql-master-ha/wiki/TableOfContents

[root@root bin]# cat /etc/masterha/app1.cnf
[server default]
manager_workdir=/var/log/masterha/app1
manager_log=/var/log/masterha/app1/manager.log
master_binlog_dir=/var/lib/mysql/data
master_ip_failover_script=/usr/bin/master_ip_failover
master_ip_online_change_script=/usr/bin/master_ip_online_change
user=mha_monitor
password=mha_monitor
ping_interval=1
remote_workdir=/tmp
repl_user=repl
repl_password=repl
ssh_user=root

[server1]
hostname=192.168.10.50
#candidate_master=1
port=3306

[server2]
hostname=192.168.10.51
candidate_master=1
check_repl_delay=0
port=3306

[server3]
hostname=192.168.10.52
port=3306
[root@root bin]#

3、设置 relay_log 的清除方式，在每个 slave 节点上：

mysql> set global relay_log_purge=0;（在 2 个 slave 上执行）
Query OK, 0 rows affected (0.00 sec)
注意：
MHA 在发生切换的过程中，从库的恢复过程中依赖于 relay log 的相关信息，所以这里要将 relay log 的自动清除设置为 OFF，采用手动清除 relay log的方式。在默认情况下，从服务器上的中继日志会在 SQL 线程执行完毕后被自动删除。但是在 MHA 环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在 ext3 的文件系统下，删除大的文件需要一定的时间，会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在linux 系统中通过硬链接删除大文件速度会很快。（在 mysql 数据库中，删除大表时，通常也采用建立硬链接的方式）。MHA 节点中包含pure_relay_logs 命令工具，它可以为中继日志创建硬链接，执行 SET GLOBAL relay_log_purge=1,等待几秒钟以便 SQL 线程切换到新的中继日志，再执行 SET GLOBAL relay_log_purge=0。以下内容可自行写脚本实现清空 relay 日志，测试也可不做：

[root@root ~]# vim /usr/bin/purge_relay_logs
pure_relay_logs 脚本参数如下所示：
--user mysql 用户名
--password mysql 密码
--port 端口号
--workdir 指定创建 relay log 的硬链接的位置，默认是/var/tmp，由于系统不同分区创建硬链接文件会失败，故需要执行硬链接具体位置，成功执行脚本后，硬链接的中继日志文件被删除

--disable_relay_log_purge 默认情况下，如果relay_log_purge=1，脚本会什么都不清理，自动退出，通过设定这个参数，当 relay_log_purge=1 的情况下会relay_log_purge 设置为 0。清理 relay log 之后，最后将参数设置为 OFF。

设置定期清理 relay 脚本（两台 slave 服务器）

cat purge_relay_log.sh
#!/bin/bash
user=root
passwd=123456
port=3306
log_dir='/var/log/masterha/app1/log'
work_dir='/var/log/masterha/app1'
purge='/usr/local/bin/purge_relay_logs'
if [ ! -d $log_dir ]
then
mkdir $log_dir -p
fi
$purge --user=$user --password=$passwd
--disable_relay_log_purge --port=$port
--workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1
添加到 crontab 定期执行
# crontab -l
0 4 * * * /bin/bash /root/purge_relay_log.sh

purge_relay_logs 脚本删除中继日志不会阻塞 SQL 线程。

4、检查 SSH 配置

[root@localhost .ssh]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Mon Sep 25 23:05:39 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Sep 25 23:05:39 2017 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Mon Sep 25 23:05:39 2017 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Mon Sep 25 23:05:39 2017 - [info] Starting SSH connection tests..
Mon Sep 25 23:05:40 2017 - [debug] 
Mon Sep 25 23:05:39 2017 - [debug]  Connecting via SSH from root@172.16.101.1(172.16.101.1:22) to root@172.16.101.3(172.16.101.3:22)..
Mon Sep 25 23:05:39 2017 - [debug]   ok.
Mon Sep 25 23:05:39 2017 - [debug]  Connecting via SSH from root@172.16.101.1(172.16.101.1:22) to root@172.16.101.2(172.16.101.2:22)..
Mon Sep 25 23:05:40 2017 - [debug]   ok.
Mon Sep 25 23:05:40 2017 - [debug] 
Mon Sep 25 23:05:39 2017 - [debug]  Connecting via SSH from root@172.16.101.3(172.16.101.3:22) to root@172.16.101.1(172.16.101.1:22)..
Mon Sep 25 23:05:40 2017 - [debug]   ok.
Mon Sep 25 23:05:40 2017 - [debug]  Connecting via SSH from root@172.16.101.3(172.16.101.3:22) to root@172.16.101.2(172.16.101.2:22)..
Mon Sep 25 23:05:40 2017 - [debug]   ok.
Mon Sep 25 23:05:41 2017 - [debug] 
Mon Sep 25 23:05:40 2017 - [debug]  Connecting via SSH from root@172.16.101.2(172.16.101.2:22) to root@172.16.101.1(172.16.101.1:22)..
Warning: Permanently added '172.16.101.2' (RSA) to the list of known hosts.
Mon Sep 25 23:05:40 2017 - [debug]   ok.
Mon Sep 25 23:05:40 2017 - [debug]  Connecting via SSH from root@172.16.101.2(172.16.101.2:22) to root@172.16.101.3(172.16.101.3:22)..
Mon Sep 25 23:05:41 2017 - [debug]   ok.
Mon Sep 25 23:05:41 2017 - [info] All SSH connection tests passed successfully.

5、检查整个复制环境情况，在此之前把 mha 的脚本拷贝到/usr/bin 下去

master_ip_failover
master_ip_online_change

A：在主库 mha-server 上配置虚拟 IP：172.16.101.4/16

[root@root ~]# ifconfig eth0:1 172.16.101.4/16
如果配置错误，如何删除虚拟 ip：
[root@root bin]# ip addr del 192.168.10.53/24 dev eth0

B：在 mha-slave 上编辑 master_ip_failover 内容，注意脚本中的

VIP、网口名和 ssh_user 用户名要根据自己的设置来。

脚本要能实现虚拟 ip 自动漂移：
这里是修改/usr/ bin/master_ip_failover
修改后内容为：

#!/usr/bin/env perl
use strict;
use warnings FATAL =>'all';
use Getopt::Long;
my (
$command, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip,
$new_master_port
);
my $vip = '172.16.101.4/16'; # Virtual IP
my $key = "1";
my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";
my $exit_code = 0;
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
#print "\n\nIN SCRIPT
TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip,
$orig_master_port are passed.
# If you manage master ip address at global catalog
database,
# invalidate orig_master_ip here.
my $exit_code = 1;
eval {
print
"\n\n\n**************************************************
*************\n";
print "Disabling the VIP - $vip on old master:
$orig_master_host\n";
print
"********************************************************
*******\n\n\n\n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog
database,
# activate new_master_ip here.
# You can also grant write access (create user, set
read_only=0, etc) here.
my $exit_code = 10;
eval {
print
"\n\n\n**************************************************
*************\n";
print "Enabling the VIP - $vip on new master:
$new_master_host \n";
print
"********************************************************
*******\n\n\n\n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $ssh_user\@$orig_master_host \"
$ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover
--command=start|stop|stopssh|status
--orig_master_host=host --orig_master_ip=ip
--orig_master_port=port --new_master_host=host
--new_master_ip=ip --new_master_port=port\n";
}

C：检查复制前，修改 read-only 参数，最好是搭建主从的时候就指定好：

[root@root bin]# masterha_check_repl --conf=/etc/masterha/app1.cnf
注意：
在检查复制的时候，主上的 read_only 参数一定是 off，备主上的
read_only 一定是 on，这个参数一定要配置。
read-only 参数可以动态修改，先把他加到各自的配置文件中，再动态修改：
在主库 mha-server 上：
mysql> set global read_only=off;
Query OK, 0 rows affected (0.32 sec)
mysql>
在两个从库上：
mysql> set global read_only=on;
Query OK, 0 rows affected (0.07 sec)

D：在 mha-slave 监控机器上，检查复制的状态：

[root@localhost mysql]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Tue Sep 26 00:04:25 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Sep 26 00:04:25 2017 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Tue Sep 26 00:04:25 2017 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Tue Sep 26 00:04:25 2017 - [info] MHA::MasterMonitor version 0.55.
Tue Sep 26 00:04:26 2017 - [info] Dead Servers:
Tue Sep 26 00:04:26 2017 - [info] Alive Servers:
Tue Sep 26 00:04:26 2017 - [info]   172.16.101.1(172.16.101.1:3306)
Tue Sep 26 00:04:26 2017 - [info]   172.16.101.3(172.16.101.3:3306)
Tue Sep 26 00:04:26 2017 - [info]   172.16.101.2(172.16.101.2:3306)
Tue Sep 26 00:04:26 2017 - [info] Alive Slaves:
Tue Sep 26 00:04:26 2017 - [info]   172.16.101.3(172.16.101.3:3306)  Version=5.7.14-log (oldest major version between slaves) log-bin:enabled
Tue Sep 26 00:04:26 2017 - [info]     Replicating from 172.16.101.1(172.16.101.1:3306)
Tue Sep 26 00:04:26 2017 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Sep 26 00:04:26 2017 - [info]   172.16.101.2(172.16.101.2:3306)  Version=5.7.14-log (oldest major version between slaves) log-bin:enabled
Tue Sep 26 00:04:26 2017 - [info]     Replicating from 172.16.101.1(172.16.101.1:3306)
Tue Sep 26 00:04:26 2017 - [info] Current Alive Master: 172.16.101.1(172.16.101.1:3306)
Tue Sep 26 00:04:26 2017 - [info] Checking slave configurations..
Tue Sep 26 00:04:26 2017 - [info]  read_only=1 is not set on slave 172.16.101.3(172.16.101.3:3306).
Tue Sep 26 00:04:26 2017 - [info]  read_only=1 is not set on slave 172.16.101.2(172.16.101.2:3306).
Tue Sep 26 00:04:26 2017 - [info] Checking replication filtering settings..
Tue Sep 26 00:04:26 2017 - [info]  binlog_do_db= , binlog_ignore_db= 
Tue Sep 26 00:04:26 2017 - [info]  Replication filtering check ok.
Tue Sep 26 00:04:26 2017 - [info] Starting SSH connection tests..
Tue Sep 26 00:04:28 2017 - [info] All SSH connection tests passed successfully.
Tue Sep 26 00:04:28 2017 - [info] Checking MHA Node version..
Tue Sep 26 00:04:28 2017 - [info]  Version check ok.
Tue Sep 26 00:04:28 2017 - [info] Checking SSH publickey authentication settings on the current master..
Tue Sep 26 00:04:29 2017 - [info] HealthCheck: SSH to 172.16.101.1 is reachable.
Tue Sep 26 00:04:29 2017 - [info] Master MHA Node version is 0.54.
Tue Sep 26 00:04:29 2017 - [info] Checking recovery script configurations on the current master..
Tue Sep 26 00:04:29 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql/ --output_file=/tmp/save_binary_logs_test --manager_version=0.55 --start_file=mysql-bin.000001 
Tue Sep 26 00:04:29 2017 - [info]   Connecting to root@172.16.101.1(172.16.101.1).. 
  Creating /tmp if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /var/lib/mysql/, up to mysql-bin.000001
Tue Sep 26 00:04:29 2017 - [info] Master setting check done.
Tue Sep 26 00:04:29 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Tue Sep 26 00:04:29 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mha_monitor' --slave_host=172.16.101.3 --slave_ip=172.16.101.3 --slave_port=3306 --workdir=/tmp --target_version=5.7.14-log --manager_version=0.55 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Tue Sep 26 00:04:29 2017 - [info]   Connecting to root@172.16.101.3(172.16.101.3:22).. 
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to localhost-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/localhost-relay-bin.000002
    Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Tue Sep 26 00:04:29 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mha_monitor' --slave_host=172.16.101.2 --slave_ip=172.16.101.2 --slave_port=3306 --workdir=/tmp --target_version=5.7.14-log --manager_version=0.55 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Tue Sep 26 00:04:29 2017 - [info]   Connecting to root@172.16.101.2(172.16.101.2:22).. 
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to localhost-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/localhost-relay-bin.000002
    Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Tue Sep 26 00:04:30 2017 - [info] Slaves settings check done.
Tue Sep 26 00:04:30 2017 - [info] 
172.16.101.1 (current master)
 +--172.16.101.3
 +--172.16.101.2

Tue Sep 26 00:04:30 2017 - [info] Checking replication health on 172.16.101.3..
Tue Sep 26 00:04:30 2017 - [info]  ok.
Tue Sep 26 00:04:30 2017 - [info] Checking replication health on 172.16.101.2..
Tue Sep 26 00:04:30 2017 - [info]  ok.
Tue Sep 26 00:04:30 2017 - [info] Checking master_ip_failover_script status:
Tue Sep 26 00:04:30 2017 - [info]   /usr/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=172.16.101.1 --orig_master_ip=172.16.101.1 --orig_master_port=3306 
Checking the Status of the script.. OK 
Tue Sep 26 00:04:30 2017 - [info]  OK.
Tue Sep 26 00:04:30 2017 - [warning] shutdown_script is not defined.
Tue Sep 26 00:04:30 2017 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

6、检查 MHA manager 节点的状态：

通过 master_check_status 脚本查看 manager 的状态：
[root@root bin]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).
[root@root bin]#
注意：如果正常，会显示“PING_OK”，否则会显示“NOT_RUNNING”，这代表MHA 监控没有开启。

7、开启 MHA manager 监控：

[root@root bin]# nohup masterha_manager  --conf=/etc/masterha/app1.cnf  --remove_dead_master --ignore_last_failover <  /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
[2] 1288
[root@root bin]#
另开一个窗口，查看监控日志：
[root@root ~]# tail -f /var/log/masterha/app1/manager.log
[root@root bin]# masterha_check_status
--conf=/etc/masterha/app1.cnf
app1 (pid:1288) is running(0:PING_OK), master:192.168.10.50
[root@root bin]#
查看 MHA manager 监控是否正常：
[root@root ~]# masterha_check_status  --conf=/etc/masterha/app1.cnf
app1 (pid:1288) is running(0:PING_OK), master:192.168.10.50
[1]+ Exit 1 nohup
masterha_manager --conf=/etc/masterha/app1.cnf  --remove_dead_master --ignore_last_failover < /dev/null >  /var/log/masterha/app1/manager.log 2>&1
[root@root ~]# masterha_check_status  --conf=/etc/masterha/app1.cnf
app1 (pid:1288) is running(0:PING_OK), master:192.168.10.50
[root@root ~]#

8、如何关闭 MHA manager 监控

[root@root ~]# masterha_stop --conf=/etc/masterha/app1.cnf
Stopped app1 successfully.
[root@root ~]#
查看是否关闭：
[root@root ~]# masterha_check_status  --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING). 开启 MHA 监控：
[root@root ~]# nohup masterha_manager  --conf=/etc/masterha/app1.cnf  --remove_dead_master --ignore_last_failover <  /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
[1] 10264
检查 MHA 监控状态：
[root@root ~]# masterha_check_status  --conf=/etc/masterha/app1.cnf
app1 monitoring program is now on initialization  phase(10:INITIALIZING_MONITOR). Wait for a while and try  checking again.
[root@root ~]# masterha_check_status  --conf=/etc/masterha/app1.cnf
app1 (pid:10264) is running(0:PING_OK),
master:192.168.10.50
[root@root ~]#
这时候建立快照的话，会出现 mha-server 断掉，mha-slave 机器上的 监控自动切断，需要手工连接监控，手工对 mha-server 机器重新 change master to 指向新的主机 mha-bak，具体操作见四步骤中的修复宕机的 master，需要注意宕机的 master 为了保持主从间的一致性只能做从库

测试

service mysqld stop现在的主机

slave在server宕机前

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 172.16.101.1
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 635
               Relay_Log_File: localhost-relay-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 

slave在server宕机后

mysql>  show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 172.16.101.3
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 635
               Relay_Log_File: localhost-relay-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 

MHA监控日志

----- Failover Report -----

app1: MySQL Master failover 172.16.101.1 to 172.16.101.3 succeeded

Master 172.16.101.1 is down!

Check MHA Manager logs at localhost.localdomain:/var/log/masterha/app1/manager.log for details.

Started automated(non-interactive) failover.
Invalidated master IP address on 172.16.101.1.
The latest slave 172.16.101.3(172.16.101.3:3306) has all relay logs for recovery.
Selected 172.16.101.3 as a new master.
172.16.101.3: OK: Applying all logs succeeded.
172.16.101.3: OK: Activated master IP address.
172.16.101.2: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
172.16.101.2: OK: Applying all logs succeeded. Slave started, replicating from 172.16.101.3.
172.16.101.3: Resetting slave info succeeded.
Master failover to 172.16.101.3(172.16.101.3:3306) completed successfully.