一:MHA的部署参考:简单粗暴的MHA部署
二:MHA结合Keepalived以使MySQL高可用对应用透明
1.Keepalived的Download&INSTALL
Download Keepalived
INSTALL For Keepalived
tar zxvf keepalived.xxx
cd keepalivedxxx
./configure --prefix=/usr/local/keepalived
make && make install
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
解决:yum install openssl-devel
2.Keepalived的configuration
[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@db-two keepalived-1.2.20]# mkdir /etc/keepalived
[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@db-two keepalived-1.2.20]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
vim /etc/keepalived/keepalived.conf
MASTER
[root@db-three keepalived-1.2.20]# cat /etc/keepalived/keepalived.conf
global_defs {
router_id mysql-master #修改为自己的主机名
notification_email {
gyc1412@163.com #接收邮件,可以有多个,一行一个
}
#当主、备份设备发生改变时,通过邮件通知
notification_email_from coral1213@126.com
#发送邮箱服务器
smtp_server stmp.163.com
#发送邮箱超时时间
smtp_connect_timeout 30
}
##################第一部分###################
vrrp_instance VI_1 {
state BACKUP #都修改成BACKUP
interface eth0 #绑定的网卡
virtual_router_id 60 #默认51 主从都修改为60
priority 100 #优先级,在mysql-slave上LVS上修改成80,即master>slave
advert_int 1
nopreempt #不抢占资源,意思就是它活了之后也不会再把主抢回来
authentication {
# 认证方式,可以是PASS或AH两种认证方式
auth_type PASS
# 认证密码
auth_pass 1111
}
virtual_ipaddress {
192.168.1.100
}
}
##################第二部分###################
virtual_server 192.168.1.100 3306 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.1.30 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
connect_port 3306
}
}
}
SLAVE
[root@db-two keepalived-1.2.20]# cat /etc/keepalived/keepalived.conf
global_defs {
router_id mysql-master #修改为自己的主机名
notification_email {
gyc1412@163.com #接收邮件,可以有多个,一行一个
}
#当主、备份设备发生改变时,通过邮件通知
notification_email_from coral1213@126.com
#发送邮箱服务器
smtp_server stmp.163.com
#发送邮箱超时时间
smtp_connect_timeout 30
}
##################第一部分###################
vrrp_instance VI_1 {
state BACKUP #都修改成BACKUP
interface eth0 #绑定的网卡
virtual_router_id 60 #默认51 主从都修改为60
priority 80 #优先级,在mysql-slave上LVS上修改成80
advert_int 1
nopreempt #不抢占资源,意思就是它活了之后也不会再把主抢回来
authentication {
# 认证方式,可以是PASS或AH两种认证方式
auth_type PASS
# 认证密码
auth_pass 1111
}
virtual_ipaddress {
192.168.1.100
}
}
##################第二部分###################
virtual_server 192.168.1.100 3306 {
delay_loop 6
lb_algo wrr
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.1.20 3306 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
connect_port 3306
}
}
}
Manager For Keepalived
/etc/init.d/keepalived start
/etc/init.d/keepalived stop
/etc/init.d/keepalived status
Keepalived的日志在/var/log/messages
注意:Keepalived的启动顺序对VIP绑定的影响?
3.对简单粗暴的MHA部署 mha.conf修改如下
[server default]
manager_workdir=/usr/local/mha
master_ip_failover_script=/usr/local/mha/script/master_ip_failover #master_ip_failover、master_ip_online_change、send_report脚本是由mha4mysql-manager源码包中sample/script下拷贝的
master_ip_online_change_script=/usr/local/mha/script/master_ip_online_change
password=123
remote_workdir=/usr/local/mha
repl_password=passwd
repl_user=slave
report_script=/usr/local/mha/script/send_report
shutdown_script=""
ssh_user=root
user=root
[server1]
candidate_master=1
check_repl_delay=0
hostname=192.168.1.20
[server2]
hostname=192.168.1.30
[server3]
hostname=192.168.1.40
4.对mha.conf中提到的master_ip_failover脚本修改如下
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.1.100';
my $ssh_start_vip = "/etc/init.d/keepalived start";
my $ssh_stop_vip = "/etc/init.d/keepalived stop";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
#`ssh $ssh_user\@cluster1 \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
return 0 unless ($ssh_user);
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
三:MHA+Keepalived的Failover测试以及相关日志信息如下
[root@db-three keepalived-1.2.20]# /etc/init.d/mysqld stop
Stopping mysqld: [ OK ]
[root@db-three keepalived-1.2.20]# tail -f /var/log/messages
Jul 7 11:43:39 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) Entering MASTER STATE
Jul 7 11:43:39 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) setting protocol VIPs.
Jul 7 11:43:39 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100
Jul 7 11:43:39 db-three Keepalived_healthcheckers[8271]: Netlink reflector reports IP 192.168.1.100 added
Jul 7 11:43:44 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100
Jul 7 11:56:23 db-three Keepalived_healthcheckers[8271]: TCP connection to [192.168.1.30]:3306 failed.
Jul 7 11:56:24 db-three Keepalived_healthcheckers[8271]: TCP connection to [192.168.1.30]:3306 failed.
Jul 7 11:56:24 db-three Keepalived_healthcheckers[8271]: Check on service [192.168.1.30]:3306 failed after 1 retry.
Jul 7 11:56:24 db-three Keepalived_healthcheckers[8271]: Removing service [192.168.1.30]:3306 from VS [192.168.1.100]:3306
Jul 7 11:56:24 db-three Keepalived_healthcheckers[8271]: Lost quorum 1-0=1 > 0 for VS [192.168.1.100]:3306
Jul 7 11:56:38 db-three Keepalived[8269]: Stopping
Jul 7 11:56:38 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) sent 0 priority
Jul 7 11:56:38 db-three Keepalived_vrrp[8272]: VRRP_Instance(VI_1) removing protocol VIPs.
Jul 7 11:56:38 db-three Keepalived_healthcheckers[8271]: Netlink reflector reports IP 192.168.1.100 removed
Jul 7 11:56:38 db-three Keepalived_healthcheckers[8271]: Stopped
Jul 7 11:56:39 db-three Keepalived_vrrp[8272]: Stopped
Jul 7 11:56:39 db-three Keepalived[8269]: Stopped Keepalived v1.2.20 (07/07,2016)
slave的keepalived日志信息
[root@db-two keepalived-1.2.20]# tail -f /var/log/messages
Jun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Unknown keyword 'nat_mask'
Jun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Unknown keyword 'nb_get_retry'
Jun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Using LinkWatch kernel netlink reflector...
Jun 21 15:05:37 db-two Keepalived_healthcheckers[10358]: Activating healthchecker for service [192.168.1.20]:3306
Jun 21 15:15:13 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jun 21 15:15:14 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Entering MASTER STATE
Jun 21 15:15:14 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) setting protocol VIPs.
Jun 21 15:15:14 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100
Jun 21 15:15:14 db-two Keepalived_healthcheckers[10358]: Netlink reflector reports IP 192.168.1.100 added
Jun 21 15:15:19 db-two Keepalived_vrrp[10359]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.100
mha的manager日志信息
Tue Jun 21 22:56:55 2016 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Tue Jun 21 22:56:55 2016 - [info] Executing SSH check script: exit 0
Tue Jun 21 22:56:55 2016 - [info] HealthCheck: SSH to 192.168.1.30 is reachable.
Tue Jun 21 22:56:58 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Jun 21 22:56:58 2016 - [warning] Connection failed 2 time(s)..
Tue Jun 21 22:57:01 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Jun 21 22:57:01 2016 - [warning] Connection failed 3 time(s)..
Tue Jun 21 22:57:04 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Jun 21 22:57:04 2016 - [warning] Connection failed 4 time(s)..
Tue Jun 21 22:57:04 2016 - [warning] Master is not reachable from health checker!
Tue Jun 21 22:57:04 2016 - [warning] Master 192.168.1.30(192.168.1.30:3306) is not reachable!
Tue Jun 21 22:57:04 2016 - [warning] SSH is reachable.
Tue Jun 21 22:57:04 2016 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/mha.conf again, and trying to connect to all serverrver status..
Tue Jun 21 22:57:04 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Jun 21 22:57:04 2016 - [info] Reading application default configuration from /etc/mha/mha.conf..
Tue Jun 21 22:57:04 2016 - [info] Reading server configuration from /etc/mha/mha.conf..
Tue Jun 21 22:57:04 2016 - [info] GTID failover mode = 1
Tue Jun 21 22:57:04 2016 - [info] Dead Servers:
Tue Jun 21 22:57:04 2016 - [info] 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:04 2016 - [info] Alive Servers:
Tue Jun 21 22:57:04 2016 - [info] 192.168.1.20(192.168.1.20:3306)
Tue Jun 21 22:57:04 2016 - [info] 192.168.1.40(192.168.1.40:3306)
Tue Jun 21 22:57:04 2016 - [info] Alive Slaves:
Tue Jun 21 22:57:04 2016 - [info] 192.168.1.20(192.168.1.20:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:04 2016 - [info] GTID ON
Tue Jun 21 22:57:04 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:04 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 21 22:57:04 2016 - [info] 192.168.1.40(192.168.1.40:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:04 2016 - [info] GTID ON
Tue Jun 21 22:57:04 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:04 2016 - [info] Checking slave configurations..
Tue Jun 21 22:57:04 2016 - [info] read_only=1 is not set on slave 192.168.1.20(192.168.1.20:3306).
Tue Jun 21 22:57:04 2016 - [info] read_only=1 is not set on slave 192.168.1.40(192.168.1.40:3306).
Tue Jun 21 22:57:04 2016 - [info] Checking replication filtering settings..
Tue Jun 21 22:57:04 2016 - [info] Replication filtering check ok.
Tue Jun 21 22:57:04 2016 - [info] Master is down!
Tue Jun 21 22:57:04 2016 - [info] Terminating monitoring script.
Tue Jun 21 22:57:04 2016 - [info] Got exit code 20 (Master dead).
Tue Jun 21 22:57:04 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Jun 21 22:57:04 2016 - [info] Reading application default configuration from /etc/mha/mha.conf..
Tue Jun 21 22:57:04 2016 - [info] Reading server configuration from /etc/mha/mha.conf..
Tue Jun 21 22:57:04 2016 - [info] MHA::MasterFailover version 0.56.
Tue Jun 21 22:57:04 2016 - [info] Starting master failover.
Tue Jun 21 22:57:04 2016 - [info]
Tue Jun 21 22:57:04 2016 - [info] * Phase 1: Configuration Check Phase..
Tue Jun 21 22:57:04 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] GTID failover mode = 1
Tue Jun 21 22:57:05 2016 - [info] Dead Servers:
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info] Checking master reachability via MySQL(double check)...
Tue Jun 21 22:57:05 2016 - [info] ok.
Tue Jun 21 22:57:05 2016 - [info] Alive Servers:
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.20(192.168.1.20:3306)
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.40(192.168.1.40:3306)
Tue Jun 21 22:57:05 2016 - [info] Alive Slaves:
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.20(192.168.1.20:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:05 2016 - [info] GTID ON
Tue Jun 21 22:57:05 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.40(192.168.1.40:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:05 2016 - [info] GTID ON
Tue Jun 21 22:57:05 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info] Starting GTID based failover.
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Jun 21 22:57:05 2016 - [info] Executing master IP deactivation script:
Tue Jun 21 22:57:05 2016 - [info] /usr/local/mha/script/master_ip_failover --orig_master_host=192.168.1.30 --orig_master_ip=192.168.1.30 --orig_master_port=3306 --command=stopssh --ssh_us
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Disabling the VIP on old master: 192.168.1.30
Tue Jun 21 22:57:05 2016 - [info] done.
Tue Jun 21 22:57:05 2016 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Jun 21 22:57:05 2016 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 3: Master Recovery Phase..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] The latest binary log file/position on all slaves is mysql-bin.000002:373
Tue Jun 21 22:57:05 2016 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.20(192.168.1.20:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:05 2016 - [info] GTID ON
Tue Jun 21 22:57:05 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.40(192.168.1.40:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:05 2016 - [info] GTID ON
Tue Jun 21 22:57:05 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info] The oldest binary log file/position on all slaves is mysql-bin.000002:373
Tue Jun 21 22:57:05 2016 - [info] Oldest slaves:
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.20(192.168.1.20:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:05 2016 - [info] GTID ON
Tue Jun 21 22:57:05 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.40(192.168.1.40:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:05 2016 - [info] GTID ON
Tue Jun 21 22:57:05 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 3.3: Determining New Master Phase..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] Searching new master from slaves..
Tue Jun 21 22:57:05 2016 - [info] Candidate masters from the configuration file:
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.20(192.168.1.20:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled
Tue Jun 21 22:57:05 2016 - [info] GTID ON
Tue Jun 21 22:57:05 2016 - [info] Replicating from 192.168.1.30(192.168.1.30:3306)
Tue Jun 21 22:57:05 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 21 22:57:05 2016 - [info] Non-candidate masters:
Tue Jun 21 22:57:05 2016 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Tue Jun 21 22:57:05 2016 - [info] New master is 192.168.1.20(192.168.1.20:3306)
Tue Jun 21 22:57:05 2016 - [info] Starting master failover..
Tue Jun 21 22:57:05 2016 - [info]
From:
192.168.1.30(192.168.1.30:3306) (current master)
+--192.168.1.20(192.168.1.20:3306)
+--192.168.1.40(192.168.1.40:3306)
To:
192.168.1.20(192.168.1.20:3306) (new master)
+--192.168.1.40(192.168.1.40:3306)
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 3.3: New Master Recovery Phase..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] Waiting all logs to be applied..
Tue Jun 21 22:57:05 2016 - [info] done.
Tue Jun 21 22:57:05 2016 - [info] Getting new master's binlog name and position..
Tue Jun 21 22:57:05 2016 - [info] master-bin.000004:231
Tue Jun 21 22:57:05 2016 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.20', MASTER_PORT=3306, MASTER_AUTO_POSIT_USER='slave', MASTER_PASSWORD='xxx';
Tue Jun 21 22:57:05 2016 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: master-bin.000004, 231, 252b2aad-4399-11e6-8ba8-000c29ca702d:2,
4a81183e-3722-11e6-ba60-000c29443328:1-3
Tue Jun 21 22:57:05 2016 - [info] Executing master IP activate script:
Tue Jun 21 22:57:05 2016 - [info] /usr/local/mha/script/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.30 --orig_master_ip=192.168.1.30 --orig_master_portaster_host=192.168.1.20 --new_master_ip=192.168.1.20 --new_master_port=3306 --new_master_user='root' --new_master_password='123'
Unknown option: new_master_user
Unknown option: new_master_password
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Enabling the VIP - 192.168.1.100 on the new master - 192.168.1.20
Tue Jun 21 22:57:05 2016 - [info] OK.
Tue Jun 21 22:57:05 2016 - [info] ** Finished master recovery successfully.
Tue Jun 21 22:57:05 2016 - [info] * Phase 3: Master Recovery Phase completed.
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 4: Slaves Recovery Phase..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 4.1: Starting Slaves in parallel..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] -- Slave recovery on host 192.168.1.40(192.168.1.40:3306) started, pid: 37397. Check tmp log /usr/local/mha/192.168.1.40_3306_20160621225704.log if it take
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] Log messages from 192.168.1.40 ...
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] Resetting slave 192.168.1.40(192.168.1.40:3306) and starting replication from the new master 192.168.1.20(192.168.1.20:3306)..
Tue Jun 21 22:57:05 2016 - [info] Executed CHANGE MASTER.
Tue Jun 21 22:57:05 2016 - [info] Slave started.
Tue Jun 21 22:57:05 2016 - [info] gtid_wait(252b2aad-4399-11e6-8ba8-000c29ca702d:2,
4a81183e-3722-11e6-ba60-000c29443328:1-3) completed on 192.168.1.40(192.168.1.40:3306). Executed 0 events.
Tue Jun 21 22:57:05 2016 - [info] End of log messages from 192.168.1.40.
Tue Jun 21 22:57:05 2016 - [info] -- Slave on host 192.168.1.40(192.168.1.40:3306) started.
Tue Jun 21 22:57:05 2016 - [info] All new slave servers recovered successfully.
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] * Phase 5: New master cleanup phase..
Tue Jun 21 22:57:05 2016 - [info]
Tue Jun 21 22:57:05 2016 - [info] Resetting slave info on the new master..
Tue Jun 21 22:57:05 2016 - [info] 192.168.1.20: Resetting slave info succeeded.
Tue Jun 21 22:57:05 2016 - [info] Master failover to 192.168.1.20(192.168.1.20:3306) completed successfully.
Tue Jun 21 22:57:05 2016 - [info] Deleted server2 entry from /etc/mha/mha.conf .
Tue Jun 21 22:57:05 2016 - [info]
----- Failover Report -----
mha: MySQL Master failover 192.168.1.30(192.168.1.30:3306) to 192.168.1.20(192.168.1.20:3306) succeeded
Master 192.168.1.30(192.168.1.30:3306) is down!
Check MHA Manager logs at db-one for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.1.30(192.168.1.30:3306)
Selected 192.168.1.20(192.168.1.20:3306) as a new master.
192.168.1.20(192.168.1.20:3306): OK: Applying all logs succeeded.
192.168.1.20(192.168.1.20:3306): OK: Activated master IP address.
192.168.1.40(192.168.1.40:3306): OK: Slave started, replicating from 192.168.1.20(192.168.1.20:3306)
192.168.1.20(192.168.1.20:3306): Resetting slave info succeeded.
Master failover to 192.168.1.20(192.168.1.20:3306) completed successfully.
Tue Jun 21 22:57:05 2016 - [info] Sending mail..
(over)