docker mysql mha_heal/mysql-mha-docker

mysql-mha-docker git:(master) ✗ sh ./scripts/mha_start_manager.sh

Sat Jun 13 14:51:29 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sat Jun 13 14:51:29 2020 - [info] Reading application default configuration from /etc/mha/app1.conf..

Sat Jun 13 14:51:29 2020 - [info] Reading server configuration from /etc/mha/app1.conf..

Sat Jun 13 14:51:29 2020 - [info] MHA::MasterMonitor version 0.57.

Sat Jun 13 14:51:29 2020 - [info] GTID failover mode = 0

Sat Jun 13 14:51:29 2020 - [info] Dead Servers:

Sat Jun 13 14:51:29 2020 - [info] Alive Servers:

Sat Jun 13 14:51:29 2020 - [info] 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:51:29 2020 - [info] 10.5.0.11(10.5.0.11:3306)

Sat Jun 13 14:51:29 2020 - [info] 10.5.0.12(10.5.0.12:3306)

Sat Jun 13 14:51:29 2020 - [info] Alive Slaves:

Sat Jun 13 14:51:29 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:51:29 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:51:29 2020 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Jun 13 14:51:29 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:51:29 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:51:29 2020 - [info] Not candidate for the new Master (no_master is set)

Sat Jun 13 14:51:29 2020 - [info] Current Alive Master: 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:51:29 2020 - [info] Checking slave configurations..

Sat Jun 13 14:51:29 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.11(10.5.0.11:3306).

Sat Jun 13 14:51:29 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.12(10.5.0.12:3306).

Sat Jun 13 14:51:29 2020 - [info] Checking replication filtering settings..

Sat Jun 13 14:51:29 2020 - [info] binlog_do_db= , binlog_ignore_db= information_schema,performance_schema,sys

Sat Jun 13 14:51:29 2020 - [info] Replication filtering check ok.

Sat Jun 13 14:51:29 2020 - [info] GTID (with auto-pos) is not supported

Sat Jun 13 14:51:29 2020 - [info] Starting SSH connection tests..

Sat Jun 13 14:51:31 2020 - [info] All SSH connection tests passed successfully.

Sat Jun 13 14:51:31 2020 - [info] Checking MHA Node version..

Sat Jun 13 14:51:31 2020 - [info] Version check ok.

Sat Jun 13 14:51:31 2020 - [info] Checking SSH publickey authentication settings on the current master..

Sat Jun 13 14:51:31 2020 - [info] HealthCheck: SSH to 10.5.0.10 is reachable.

Sat Jun 13 14:51:31 2020 - [info] Master MHA Node version is 0.57.

Sat Jun 13 14:51:31 2020 - [info] Checking recovery script configurations on 10.5.0.10(10.5.0.10:3306)..

Sat Jun 13 14:51:31 2020 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000001

Sat Jun 13 14:51:31 2020 - [info] Connecting to root@10.5.0.10(10.5.0.10:22)..

Creating /usr/local/mha if not exists.. ok.

Checking output directory is accessible or not..

ok.

Binlog found at /var/lib/mysql, up to mysql-bin.000001

Sat Jun 13 14:51:31 2020 - [info] Binlog setting check done.

Sat Jun 13 14:51:31 2020 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Sat Jun 13 14:51:31 2020 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.5.0.11 --slave_ip=10.5.0.11 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.7.19-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Jun 13 14:51:31 2020 - [info] Connecting to root@10.5.0.11(10.5.0.11:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to mysql-relay-bin.000004

Temporary relay log file is /var/lib/mysql/mysql-relay-bin.000004

Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.

done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Jun 13 14:51:32 2020 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.5.0.12 --slave_ip=10.5.0.12 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.7.19-log --manager_version=0.57 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx

Sat Jun 13 14:51:32 2020 - [info] Connecting to root@10.5.0.12(10.5.0.12:22)..

Checking slave recovery environment settings..

Opening /var/lib/mysql/relay-log.info ... ok.

Relay log found at /var/lib/mysql, up to mysql-relay-bin.000008

Temporary relay log file is /var/lib/mysql/mysql-relay-bin.000008

Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.

done.

Testing mysqlbinlog output.. done.

Cleaning up test file(s).. done.

Sat Jun 13 14:51:32 2020 - [info] Slaves settings check done.

Sat Jun 13 14:51:32 2020 - [info]

10.5.0.10(10.5.0.10:3306) (current master)

+--10.5.0.11(10.5.0.11:3306)

+--10.5.0.12(10.5.0.12:3306)

Sat Jun 13 14:51:32 2020 - [warning] master_ip_failover_script is not defined.

Sat Jun 13 14:51:32 2020 - [warning] shutdown_script is not defined.

Sat Jun 13 14:51:32 2020 - [info] Set master ping interval 5 seconds.

Sat Jun 13 14:51:32 2020 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.

Sat Jun 13 14:51:32 2020 - [info] Starting ping health check on 10.5.0.10(10.5.0.10:3306)..

Sat Jun 13 14:51:32 2020 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

Sat Jun 13 14:51:52 2020 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)

Sat Jun 13 14:51:52 2020 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin

Connection to 10.5.0.10 closed by remote host.

Sat Jun 13 14:51:52 2020 - [warning] HealthCheck: SSH to 10.5.0.10 is NOT reachable.

Sat Jun 13 14:52:02 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.5.0.10' (4))

Sat Jun 13 14:52:02 2020 - [warning] Connection failed 2 time(s)..

Sat Jun 13 14:52:07 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.5.0.10' (4))

Sat Jun 13 14:52:07 2020 - [warning] Connection failed 3 time(s)..

Sat Jun 13 14:52:12 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.5.0.10' (4))

Sat Jun 13 14:52:12 2020 - [warning] Connection failed 4 time(s)..

Sat Jun 13 14:52:12 2020 - [warning] Master is not reachable from health checker!

Sat Jun 13 14:52:12 2020 - [warning] Master 10.5.0.10(10.5.0.10:3306) is not reachable!

Sat Jun 13 14:52:12 2020 - [warning] SSH is NOT reachable.

Sat Jun 13 14:52:12 2020 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1.conf again, and trying to connect to all servers to check server status..

Sat Jun 13 14:52:12 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sat Jun 13 14:52:12 2020 - [info] Reading application default configuration from /etc/mha/app1.conf..

Sat Jun 13 14:52:12 2020 - [info] Reading server configuration from /etc/mha/app1.conf..

Sat Jun 13 14:52:12 2020 - [info] GTID failover mode = 0

Sat Jun 13 14:52:12 2020 - [info] Dead Servers:

Sat Jun 13 14:52:12 2020 - [info] 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:12 2020 - [info] Alive Servers:

Sat Jun 13 14:52:12 2020 - [info] 10.5.0.11(10.5.0.11:3306)

Sat Jun 13 14:52:12 2020 - [info] 10.5.0.12(10.5.0.12:3306)

Sat Jun 13 14:52:12 2020 - [info] Alive Slaves:

Sat Jun 13 14:52:12 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:12 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:12 2020 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Jun 13 14:52:12 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:12 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:12 2020 - [info] Not candidate for the new Master (no_master is set)

Sat Jun 13 14:52:12 2020 - [info] Checking slave configurations..

Sat Jun 13 14:52:12 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.11(10.5.0.11:3306).

Sat Jun 13 14:52:12 2020 - [warning] relay_log_purge=0 is not set on slave 10.5.0.12(10.5.0.12:3306).

Sat Jun 13 14:52:12 2020 - [info] Checking replication filtering settings..

Sat Jun 13 14:52:12 2020 - [info] Replication filtering check ok.

Sat Jun 13 14:52:12 2020 - [info] Master is down!

Sat Jun 13 14:52:12 2020 - [info] Terminating monitoring script.

Sat Jun 13 14:52:12 2020 - [info] Got exit code 20 (Master dead).

Sat Jun 13 14:52:12 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sat Jun 13 14:52:12 2020 - [info] Reading application default configuration from /etc/mha/app1.conf..

Sat Jun 13 14:52:12 2020 - [info] Reading server configuration from /etc/mha/app1.conf..

Sat Jun 13 14:52:12 2020 - [info] MHA::MasterFailover version 0.57.

Sat Jun 13 14:52:12 2020 - [info] Starting master failover.

Sat Jun 13 14:52:12 2020 - [info]

Sat Jun 13 14:52:12 2020 - [info] * Phase 1: Configuration Check Phase..

Sat Jun 13 14:52:12 2020 - [info]

Sat Jun 13 14:52:12 2020 - [info] GTID failover mode = 0

Sat Jun 13 14:52:12 2020 - [info] Dead Servers:

Sat Jun 13 14:52:12 2020 - [info] 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:12 2020 - [info] Checking master reachability via MySQL(double check)...

Sat Jun 13 14:52:13 2020 - [info] ok.

Sat Jun 13 14:52:13 2020 - [info] Alive Servers:

Sat Jun 13 14:52:13 2020 - [info] 10.5.0.11(10.5.0.11:3306)

Sat Jun 13 14:52:13 2020 - [info] 10.5.0.12(10.5.0.12:3306)

Sat Jun 13 14:52:13 2020 - [info] Alive Slaves:

Sat Jun 13 14:52:13 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:13 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:13 2020 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Jun 13 14:52:13 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:13 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:13 2020 - [info] Not candidate for the new Master (no_master is set)

Sat Jun 13 14:52:13 2020 - [info] Starting Non-GTID based failover.

Sat Jun 13 14:52:13 2020 - [info]

Sat Jun 13 14:52:13 2020 - [info] ** Phase 1: Configuration Check Phase completed.

Sat Jun 13 14:52:13 2020 - [info]

Sat Jun 13 14:52:13 2020 - [info] * Phase 2: Dead Master Shutdown Phase..

Sat Jun 13 14:52:13 2020 - [info]

Sat Jun 13 14:52:13 2020 - [info] Forcing shutdown so that applications never connect to the current master..

Sat Jun 13 14:52:13 2020 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.

Sat Jun 13 14:52:13 2020 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.

Sat Jun 13 14:52:14 2020 - [info] * Phase 2: Dead Master Shutdown Phase completed.

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 3: Master Recovery Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 3.1: Getting Latest Slaves Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] The latest binary log file/position on all slaves is mysql-bin.000001:154

Sat Jun 13 14:52:14 2020 - [info] Latest slaves (Slaves that received relay log files to the latest):

Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:14 2020 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Jun 13 14:52:14 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:14 2020 - [info] Not candidate for the new Master (no_master is set)

Sat Jun 13 14:52:14 2020 - [info] The oldest binary log file/position on all slaves is mysql-bin.000001:154

Sat Jun 13 14:52:14 2020 - [info] Oldest slaves:

Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:14 2020 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Jun 13 14:52:14 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:14 2020 - [info] Not candidate for the new Master (no_master is set)

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [warning] Dead Master is not SSH reachable. Could not save it's binlogs. Transactions that were not sent to the latest slave (Read_Master_Log_Pos to the tail of the dead master's binlog) were lost.

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 3.3: Determining New Master Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] Finding the latest slave that has all relay logs for recovering other slaves..

Sat Jun 13 14:52:14 2020 - [info] All slaves received relay logs to the same position. No need to resync each other.

Sat Jun 13 14:52:14 2020 - [info] Searching new master from slaves..

Sat Jun 13 14:52:14 2020 - [info] Candidate masters from the configuration file:

Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11(10.5.0.11:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:14 2020 - [info] Primary candidate for the new Master (candidate_master is set)

Sat Jun 13 14:52:14 2020 - [info] Non-candidate masters:

Sat Jun 13 14:52:14 2020 - [info] 10.5.0.12(10.5.0.12:3306) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sat Jun 13 14:52:14 2020 - [info] Replicating from 10.5.0.10(10.5.0.10:3306)

Sat Jun 13 14:52:14 2020 - [info] Not candidate for the new Master (no_master is set)

Sat Jun 13 14:52:14 2020 - [info] Searching from candidate_master slaves which have received the latest relay log events..

Sat Jun 13 14:52:14 2020 - [info] New master is 10.5.0.11(10.5.0.11:3306)

Sat Jun 13 14:52:14 2020 - [info] Starting master failover..

Sat Jun 13 14:52:14 2020 - [info]

From:

10.5.0.10(10.5.0.10:3306) (current master)

+--10.5.0.11(10.5.0.11:3306)

+--10.5.0.12(10.5.0.12:3306)

To:

10.5.0.11(10.5.0.11:3306) (new master)

+--10.5.0.12(10.5.0.12:3306)

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 3.3: New Master Diff Log Generation Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. No need to generate diff files from the latest slave.

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 3.4: Master Log Apply Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.

Sat Jun 13 14:52:14 2020 - [info] Starting recovery on 10.5.0.11(10.5.0.11:3306)..

Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. Waiting all logs to be applied..

Sat Jun 13 14:52:14 2020 - [info] done.

Sat Jun 13 14:52:14 2020 - [info] All relay logs were successfully applied.

Sat Jun 13 14:52:14 2020 - [info] Getting new master's binlog name and position..

Sat Jun 13 14:52:14 2020 - [info] mysql-bin.000001:443

Sat Jun 13 14:52:14 2020 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.5.0.11', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=443, MASTER_USER='myslave', MASTER_PASSWORD='xxx';

Sat Jun 13 14:52:14 2020 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.

Sat Jun 13 14:52:14 2020 - [info] Setting read_only=0 on 10.5.0.11(10.5.0.11:3306)..

Sat Jun 13 14:52:14 2020 - [info] ok.

Sat Jun 13 14:52:14 2020 - [info] ** Finished master recovery successfully.

Sat Jun 13 14:52:14 2020 - [info] * Phase 3: Master Recovery Phase completed.

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 4: Slaves Recovery Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] -- Slave diff file generation on host 10.5.0.12(10.5.0.12:3306) started, pid: 822. Check tmp log /usr/local/mha/10.5.0.12_3306_20200613145212.log if it takes time..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] Log messages from 10.5.0.12 ...

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. No need to generate diff files from the latest slave.

Sat Jun 13 14:52:14 2020 - [info] End of log messages from 10.5.0.12.

Sat Jun 13 14:52:14 2020 - [info] -- 10.5.0.12(10.5.0.12:3306) has the latest relay log events.

Sat Jun 13 14:52:14 2020 - [info] Generating relay diff files from the latest slave succeeded.

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] -- Slave recovery on host 10.5.0.12(10.5.0.12:3306) started, pid: 824. Check tmp log /usr/local/mha/10.5.0.12_3306_20200613145212.log if it takes time..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] Log messages from 10.5.0.12 ...

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] Starting recovery on 10.5.0.12(10.5.0.12:3306)..

Sat Jun 13 14:52:14 2020 - [info] This server has all relay logs. Waiting all logs to be applied..

Sat Jun 13 14:52:14 2020 - [info] done.

Sat Jun 13 14:52:14 2020 - [info] All relay logs were successfully applied.

Sat Jun 13 14:52:14 2020 - [info] Resetting slave 10.5.0.12(10.5.0.12:3306) and starting replication from the new master 10.5.0.11(10.5.0.11:3306)..

Sat Jun 13 14:52:14 2020 - [info] Executed CHANGE MASTER.

Sat Jun 13 14:52:14 2020 - [info] Slave started.

Sat Jun 13 14:52:14 2020 - [info] End of log messages from 10.5.0.12.

Sat Jun 13 14:52:14 2020 - [info] -- Slave recovery on host 10.5.0.12(10.5.0.12:3306) succeeded.

Sat Jun 13 14:52:14 2020 - [info] All new slave servers recovered successfully.

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] * Phase 5: New master cleanup phase..

Sat Jun 13 14:52:14 2020 - [info]

Sat Jun 13 14:52:14 2020 - [info] Resetting slave info on the new master..

Sat Jun 13 14:52:14 2020 - [info] 10.5.0.11: Resetting slave info succeeded.

Sat Jun 13 14:52:14 2020 - [info] Master failover to 10.5.0.11(10.5.0.11:3306) completed successfully.

Sat Jun 13 14:52:14 2020 - [info] Deleted server0 entry from /etc/mha/app1.conf .

Sat Jun 13 14:52:14 2020 - [info]

----- Failover Report -----

app1: MySQL Master failover 10.5.0.10(10.5.0.10:3306) to 10.5.0.11(10.5.0.11:3306) succeeded

Master 10.5.0.10(10.5.0.10:3306) is down!

Check MHA Manager logs at 1a83c543536c for details.

Started automated(non-interactive) failover.

The latest slave 10.5.0.11(10.5.0.11:3306) has all relay logs for recovery.

Selected 10.5.0.11(10.5.0.11:3306) as a new master.

10.5.0.11(10.5.0.11:3306): OK: Applying all logs succeeded.

10.5.0.12(10.5.0.12:3306): This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

10.5.0.12(10.5.0.12:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.5.0.11(10.5.0.11:3306)

10.5.0.11(10.5.0.11:3306): Resetting slave info succeeded.

Master failover to 10.5.0.11(10.5.0.11:3306) completed successfully.