MHA故障转移日志

该日志在MHA管理主机中，配置文件中你所定义的日志文件中。

在认定master主库宕机之后：故障转移步骤

Thu Jul 27 19:19:56 2017 - [info] * Phase 1: Configuration Check Phase..
Thu Jul 27 19:19:56 2017 - [info] 
Thu Jul 27 19:19:56 2017 - [info] GTID failover mode = 1
Thu Jul 27 19:19:56 2017 - [info] Dead Servers:
Thu Jul 27 19:19:56 2017 - [info]   172.25.67.3(172.25.67.3:3306)
Thu Jul 27 19:19:56 2017 - [info] Checking master reachability via MySQL(double check)...
Thu Jul 27 19:19:56 2017 - [info]  ok.
Thu Jul 27 19:19:56 2017 - [info] Alive Servers:
Thu Jul 27 19:19:56 2017 - [info]   172.25.67.2(172.25.67.2:3306)
Thu Jul 27 19:19:56 2017 - [info]   172.25.67.4(172.25.67.4:3306)
Thu Jul 27 19:19:56 2017 - [info] Alive Slaves:
Thu Jul 27 19:19:56 2017 - [info]   172.25.67.2(172.25.67.2:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Thu Jul 27 19:19:56 2017 - [info]     GTID ON
Thu Jul 27 19:19:56 2017 - [info]     Replicating from 172.25.67.3(172.25.67.3:3306)
Thu Jul 27 19:19:56 2017 - [info]   172.25.67.4(172.25.67.4:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Thu Jul 27 19:19:56 2017 - [info]     GTID ON
Thu Jul 27 19:19:56 2017 - [info]     Replicating from 172.25.67.3(172.25.67.3:3306)
Thu Jul 27 19:19:56 2017 - [info] Starting GTID based failover.
Thu Jul 27 19:19:56 2017 - [info] 
Thu Jul 27 19:19:56 2017 - [info] ** Phase 1: Configuration Check Phase completed.

该第一阶段：配置检查。从上面的日志信息中可以得知死掉的主库是(172.25.67.3)，存活的从库是(172.25.67.2/4)，版本为5.7.17-log，且开启了GTID。

Thu Jul 27 19:19:56 2017 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Jul 27 19:19:56 2017 - [info] 
Thu Jul 27 19:19:56 2017 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Jul 27 19:19:56 2017 - [info] Executing master IP deactivation script:
Thu Jul 27 19:19:56 2017 - [info]   /etc/mha/master_ip_failover --orig_master_host=172.25.67.3 --orig_master_ip=172.25.67.3 --orig_master_port=3306 --command=stopssh --ssh_user=root  


IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 172.25.67.100/24===

Disabling the VIP on old master: 172.25.67.3 
SIOCSIFFLAGS: Cannot assign requested address
Thu Jul 27 19:19:57 2017 - [info]  done.
Thu Jul 27 19:19:57 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Jul 27 19:19:57 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.

该第二阶段是把挂掉的master从虚拟的VIP中移除。下面是第三阶段：

Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] * Phase 3: Master Recovery Phase..
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] The latest binary log file/position on all slaves is mysql-bin.000003:194
Thu Jul 27 19:19:57 2017 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Jul 27 19:19:57 2017 - [info]   172.25.67.2(172.25.67.2:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Thu Jul 27 19:19:57 2017 - [info]     GTID ON
Thu Jul 27 19:19:57 2017 - [info]     Replicating from 172.25.67.3(172.25.67.3:3306)
Thu Jul 27 19:19:57 2017 - [info]   172.25.67.4(172.25.67.4:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Thu Jul 27 19:19:57 2017 - [info]     GTID ON
Thu Jul 27 19:19:57 2017 - [info]     Replicating from 172.25.67.3(172.25.67.3:3306)
Thu Jul 27 19:19:57 2017 - [info] The oldest binary log file/position on all slaves is mysql-bin.000003:194
Thu Jul 27 19:19:57 2017 - [info] Oldest slaves:
Thu Jul 27 19:19:57 2017 - [info]   172.25.67.2(172.25.67.2:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Thu Jul 27 19:19:57 2017 - [info]     GTID ON
Thu Jul 27 19:19:57 2017 - [info]     Replicating from 172.25.67.3(172.25.67.3:3306)
Thu Jul 27 19:19:57 2017 - [info]   172.25.67.4(172.25.67.4:3306)  Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Thu Jul 27 19:19:57 2017 - [info]     GTID ON
Thu Jul 27 19:19:57 2017 - [info]     Replicating from 172.25.67.3(172.25.67.3:3306)

该第三阶段是进行最新的slave数据恢复。此阶段中首先要检查最新的的slave和最老的slave所接收的master上的binlog是否都为mysql-bin.000003，position位置是否为194。

Thu Jul 27 19:19:57 2017 - [info]  * Phase 3.2

然后，其实还会有3.2，用于抓取主库上未发送的binlog，并保存到对应指定的位置。但我这里本来就全同步了，所以就没有3.2。

Thu Jul 27 19:19:57 2017 - [info] * Phase 3.3: Determining New Master Phase..
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] Searching new master from slaves..
Thu Jul 27 19:19:57 2017 - [info]  Candidate masters from the configuration file:
Thu Jul 27 19:19:57 2017 - [info]  Non-candidate masters:
Thu Jul 27 19:19:57 2017 - [info] New master is 172.25.67.2(172.25.67.2:3306)
Thu Jul 27 19:19:57 2017 - [info] Starting master failover..
Thu Jul 27 19:19:57 2017 - [info] 
From:
172.25.67.3(172.25.67.3:3306) (current master)
 +--172.25.67.2(172.25.67.2:3306)
 +--172.25.67.4(172.25.67.4:3306)

To:
172.25.67.2(172.25.67.2:3306) (new master)
 +--172.25.67.4(172.25.67.4:3306)
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] * Phase 3.3: New Master Recovery Phase..
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info]  Waiting all logs to be applied.. 
Thu Jul 27 19:19:57 2017 - [info]   done.
Thu Jul 27 19:19:57 2017 - [info] Getting new master's binlog name and position..
Thu Jul 27 19:19:57 2017 - [info]  mysql-bin.000011:194
Thu Jul 27 19:19:57 2017 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='172.25.67.2', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='redhat', MASTER_PASSWORD='xxx';
Thu Jul 27 19:19:57 2017 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000011, 194, e1b8a5b6-7274-11e7-8d14-525400813e46:1-3,
e9a0c0f6-7274-11e7-8dde-525400022470:1
Thu Jul 27 19:19:57 2017 - [info] Executing master IP activate script:
Thu Jul 27 19:19:57 2017 - [info]   /etc/mha/master_ip_failover --command=start --ssh_user=root --orig_master_host=172.25.67.3 --orig_master_ip=172.25.67.3 --orig_master_port=3306 --new_master_host=172.25.67.2 --new_master_ip=172.25.67.2 --new_master_port=3306 --new_master_user='root' --new_master_password='Gmoon+007'  
Unknown option: new_master_user
Unknown option: new_master_password


IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 172.25.67.100/24===

Enabling the VIP - 172.25.67.100/24 on the new master - 172.25.67.2 
Thu Jul 27 19:19:57 2017 - [info]  OK.
Thu Jul 27 19:19:57 2017 - [info] ** Finished master recovery successfully.
Thu Jul 27 19:19:57 2017 - [info] * Phase 3: Master Recovery Phase completed.

这里提升一个新的master，其它slave都指向新的master。并通过命令master_ip_failover将故障转移，将VIP漂移到172.25.67.2上，从而使其提升为新的master。

Thu Jul 27 19:19:57 2017 - [info] * Phase 4: Slaves Recovery Phase..
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] * Phase 4.1: Starting Slaves in parallel..
Thu Jul 27 19:19:57 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info] -- Slave recovery on host 172.25.67.4(172.25.67.4:3306) started, pid: 2827. Check tmp log /usr/local/mha/172.25.67.4_3306_20170727191956.log if it takes time..
Thu Jul 27 19:20:00 2017 - [info] 
Thu Jul 27 19:20:00 2017 - [info] Log messages from 172.25.67.4 ...
Thu Jul 27 19:20:00 2017 - [info] 
Thu Jul 27 19:19:57 2017 - [info]  Resetting slave 172.25.67.4(172.25.67.4:3306) and starting replication from the new master 172.25.67.2(172.25.67.2:3306)..
Thu Jul 27 19:19:59 2017 - [info]  Executed CHANGE MASTER.
Thu Jul 27 19:20:00 2017 - [info]  Slave started.
Thu Jul 27 19:20:00 2017 - [info]  gtid_wait(e1b8a5b6-7274-11e7-8d14-525400813e46:1-3,
e9a0c0f6-7274-11e7-8dde-525400022470:1) completed on 172.25.67.4(172.25.67.4:3306). Executed 0 events.
Thu Jul 27 19:20:00 2017 - [info] End of log messages from 172.25.67.4.
Thu Jul 27 19:20:00 2017 - [info] -- Slave on host 172.25.67.4(172.25.67.4:3306) started.
Thu Jul 27 19:20:00 2017 - [info] All new slave servers recovered successfully.

这一阶段是恢复最老的slave数据，当最老的slave数据和最新的slave上的relay log补齐后，MHA将master缺失的那一部分binlog发送给最老的slave，将数据补齐，之后重置同步复制关系。

Thu Jul 27 19:20:00 2017 - [info] * Phase 5: New master cleanup phase..
Thu Jul 27 19:20:00 2017 - [info] 
Thu Jul 27 19:20:00 2017 - [info] Resetting slave info on the new master..
Thu Jul 27 19:20:01 2017 - [info]  172.25.67.2: Resetting slave info succeeded.
Thu Jul 27 19:20:01 2017 - [info] Master failover to 172.25.67.2(172.25.67.2:3306) completed successfully.
Thu Jul 27 19:20:01 2017 - [info] 

----- Failover Report -----

app: MySQL Master failover 172.25.67.3(172.25.67.3:3306) to 172.25.67.2(172.25.67.2:3306) succeeded

Master 172.25.67.3(172.25.67.3:3306) is down!

Check MHA Manager logs at server1:/usr/local/mha/mha.log for details.

Started automated(non-interactive) failover.
Invalidated master IP address on 172.25.67.3(172.25.67.3:3306)
Selected 172.25.67.2(172.25.67.2:3306) as a new master.
172.25.67.2(172.25.67.2:3306): OK: Applying all logs succeeded.
172.25.67.2(172.25.67.2:3306): OK: Activated master IP address.
172.25.67.4(172.25.67.4:3306): OK: Slave started, replicating from 172.25.67.2(172.25.67.2:3306)
172.25.67.2(172.25.67.2:3306): Resetting slave info succeeded.
Master failover to 172.25.67.2(172.25.67.2:3306) completed successfully.

最后这以阶段就是打印故障切换报告了。