mysql mha gtid_MySQL--MHA与GTID

Sun Jul 8 23:35:21 2018 - [info] MHA::MasterMonitor version 0.56.

Sun Jul8 23:35:21 2018 - [info] GTID failover mode = 1Sun Jul8 23:35:21 2018 - [info] Dead Servers:

Sun Jul8 23:35:21 2018 - [info] Alive Servers:

Sun Jul8 23:35:21 2018 - [info] 10.0.203.104(10.0.203.104:3358)

Sun Jul8 23:35:21 2018 - [info] 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:35:21 2018 - [info] 10.0.203.117(10.0.203.117:3358)

Sun Jul8 23:35:21 2018 - [info] Alive Slaves:

Sun Jul8 23:35:21 2018 - [info] 10.0.203.104(10.0.203.104:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:35:21 2018 - [info] GTID ON

Sun Jul8 23:35:21 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:35:21 2018 - [info] Primary candidate forthe new Master (candidate_master is set)

Sun Jul8 23:35:21 2018 - [info] 10.0.203.117(10.0.203.117:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:35:21 2018 - [info] GTID ON

Sun Jul8 23:35:21 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:35:21 2018 - [info] Current Alive Master: 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:35:21 2018 - [info] Checking slave configurations..

Sun Jul8 23:35:21 2018 - [info] read_only=1 is not set on slave 10.0.203.104(10.0.203.104:3358).

Sun Jul8 23:35:21 2018 - [info] read_only=1 is not set on slave 10.0.203.117(10.0.203.117:3358).

Sun Jul8 23:35:21 2018 - [info] Checking replication filtering settings..

Sun Jul8 23:35:21 2018 - [info] binlog_do_db= , binlog_ignore_db=Sun Jul8 23:35:21 2018 - [info] Replication filtering check ok.

Sun Jul8 23:35:21 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.

Sun Jul8 23:35:21 2018 - [info] Checking SSH publickey authentication settings on the current master..

Sun Jul8 23:35:22 2018 - [info] HealthCheck: SSH to 10.0.203.109is reachable.

Sun Jul8 23:35:22 2018 - [info]10.0.203.109(10.0.203.109:3358) (current master)+--10.0.203.104(10.0.203.104:3358)+--10.0.203.117(10.0.203.117:3358)

Sun Jul8 23:35:22 2018 -[warning] master_ip_failover_script is not defined.

Sun Jul8 23:35:22 2018 -[warning] shutdown_script is not defined.

Sun Jul8 23:35:22 2018 - [info] Set master ping interval 1seconds.

Sun Jul8 23:35:22 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or moreroutes.

Sun Jul8 23:35:22 2018 - [info] Starting ping health check on 10.0.203.109(10.0.203.109:3358)..

Sun Jul8 23:35:22 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

Sun Jul 8 23:35:58 2018 - [warning] Got error on MySQL select ping: 2006(MySQL server has gone away)

Sun Jul8 23:35:58 2018 - [info] Executing SSH check script: exit 0Sun Jul8 23:35:58 2018 - [info] HealthCheck: SSH to 10.0.203.109is reachable.

Sun Jul8 23:35:59 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Sun Jul8 23:35:59 2018 - [warning] Connection failed 2 time(s)..

Sun Jul8 23:36:00 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Sun Jul8 23:36:00 2018 - [warning] Connection failed 3 time(s)..

Sun Jul8 23:36:01 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)

Sun Jul8 23:36:01 2018 - [warning] Connection failed 4 time(s)..

Sun Jul8 23:36:01 2018 - [warning] Master is not reachable from health checker!Sun Jul8 23:36:01 2018 - [warning] Master 10.0.203.109(10.0.203.109:3358) is not reachable!Sun Jul8 23:36:01 2018 -[warning] SSH is reachable.

Sun Jul8 23:36:01 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..

Sun Jul8 23:36:01 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sun Jul8 23:36:01 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Sun Jul8 23:36:01 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Sun Jul8 23:36:01 2018 - [info] GTID failover mode = 1Sun Jul8 23:36:01 2018 - [info] Dead Servers:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Alive Servers:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104(10.0.203.104:3358)

Sun Jul8 23:36:01 2018 - [info] 10.0.203.117(10.0.203.117:3358)

Sun Jul8 23:36:01 2018 - [info] Alive Slaves:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104(10.0.203.104:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Primary candidate forthe new Master (candidate_master is set)

Sun Jul8 23:36:01 2018 - [info] 10.0.203.117(10.0.203.117:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Checking slave configurations..

Sun Jul8 23:36:01 2018 - [info] read_only=1 is not set on slave 10.0.203.104(10.0.203.104:3358).

Sun Jul8 23:36:01 2018 - [info] read_only=1 is not set on slave 10.0.203.117(10.0.203.117:3358).

Sun Jul8 23:36:01 2018 - [info] Checking replication filtering settings..

Sun Jul8 23:36:01 2018 - [info] Replication filtering check ok.

Sun Jul8 23:36:01 2018 - [info] Master is down!Sun Jul8 23:36:01 2018 - [info] Terminating monitoring script.

Sun Jul8 23:36:01 2018 - [info] Got exit code 20(Master dead).

Sun Jul8 23:36:01 2018 - [info] MHA::MasterFailover version 0.56.

Sun Jul8 23:36:01 2018 - [info] Starting master failover.

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 1: Configuration Check Phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] GTID failover mode = 1Sun Jul8 23:36:01 2018 - [info] Dead Servers:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Checking master reachability via MySQL(doublecheck)...

Sun Jul8 23:36:01 2018 - [info] ok.

Sun Jul8 23:36:01 2018 - [info] Alive Servers:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104(10.0.203.104:3358)

Sun Jul8 23:36:01 2018 - [info] 10.0.203.117(10.0.203.117:3358)

Sun Jul8 23:36:01 2018 - [info] Alive Slaves:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104(10.0.203.104:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Primary candidate forthe new Master (candidate_master is set)

Sun Jul8 23:36:01 2018 - [info] 10.0.203.117(10.0.203.117:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Starting GTID based failover.

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] ** Phase 1: Configuration Check Phase completed.

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 2: Dead Master Shutdown Phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] Forcing shutdown so that applications never connect to the current master..

Sun Jul8 23:36:01 2018 -[warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.

Sun Jul8 23:36:01 2018 -[warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.

Sun Jul8 23:36:01 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 3: Master Recovery Phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000008:6689Sun Jul8 23:36:01 2018 - [info] Retrieved Gtid Set: 541e0f07-8047-11e8-8434-0800270b00d2:49-69Sun Jul8 23:36:01 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104(10.0.203.104:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Primary candidate forthe new Master (candidate_master is set)

Sun Jul8 23:36:01 2018 - [info] 10.0.203.117(10.0.203.117:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000008:6689Sun Jul8 23:36:01 2018 - [info] Retrieved Gtid Set: 541e0f07-8047-11e8-8434-0800270b00d2:49-69Sun Jul8 23:36:01 2018 - [info] Oldest slaves:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104(10.0.203.104:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Primary candidate forthe new Master (candidate_master is set)

Sun Jul8 23:36:01 2018 - [info] 10.0.203.117(10.0.203.117:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 3.3: Determining New Master Phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] Searching new master from slaves..

Sun Jul8 23:36:01 2018 - [info] Candidate masters from the configuration file:

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104(10.0.203.104:3358) Version=5.7.19-log (oldest major version between slaves) log-bin:enabled

Sun Jul8 23:36:01 2018 - [info] GTID ON

Sun Jul8 23:36:01 2018 - [info] Replicating from 10.0.203.109(10.0.203.109:3358)

Sun Jul8 23:36:01 2018 - [info] Primary candidate forthe new Master (candidate_master is set)

Sun Jul8 23:36:01 2018 - [info] Non-candidate masters:

Sun Jul8 23:36:01 2018 - [info] Searching from candidate_master slaves whichhave received the latest relay log events..

Sun Jul8 23:36:01 2018 - [info] New master is 10.0.203.104(10.0.203.104:3358)

Sun Jul8 23:36:01 2018 - [info] Starting master failover..

Sun Jul8 23:36:01 2018 - [info]

From:10.0.203.109(10.0.203.109:3358) (current master)+--10.0.203.104(10.0.203.104:3358)+--10.0.203.117(10.0.203.117:3358)

To:10.0.203.104(10.0.203.104:3358) (new master)+--10.0.203.117(10.0.203.117:3358)

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 3.3: New Master Recovery Phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] Waiting all logs to be applied..

Sun Jul8 23:36:01 2018 - [info] done.

Sun Jul8 23:36:01 2018 - [info] Getting new master's binlog name and position..

Sun Jul 8 23:36:01 2018 - [info] mysql-bin.000006:77499Sun Jul8 23:36:01 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.203.104', MASTER_PORT=3358, MASTER_AUTO_POSITION=1, MASTER_USER='replicater', MASTER_PASSWORD='xxx';

Sun Jul8 23:36:01 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000006, 77499, 41d8a420-8047-11e8-8580-080027e837eb:1-92,

541e0f07-8047-11e8-8434-0800270b00d2:1-69Sun Jul8 23:36:01 2018 -[warning] master_ip_failover_script is not set. Skipping taking over new master IP address.

Sun Jul8 23:36:01 2018 - [info] **Finished master recovery successfully.

Sun Jul8 23:36:01 2018 - [info] * Phase 3: Master Recovery Phase completed.

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 4: Slaves Recovery Phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 4.1: Starting Slaves inparallel..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] -- Slave recovery on host 10.0.203.117(10.0.203.117:3358) started, pid: 5680. Check tmp log /var/log/masterha/app1/10.0.203.117_3358_20180708233601.log if it takes time..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] Log messages from 10.0.203.117...

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] Resetting slave 10.0.203.117(10.0.203.117:3358) and starting replication from the new master 10.0.203.104(10.0.203.104:3358)..

Sun Jul8 23:36:01 2018 - [info] Executed CHANGE MASTER.

Sun Jul8 23:36:01 2018 - [info] Slave started.

Sun Jul8 23:36:01 2018 - [info] gtid_wait(41d8a420-8047-11e8-8580-080027e837eb:1-92,

541e0f07-8047-11e8-8434-0800270b00d2:1-69) completed on 10.0.203.117(10.0.203.117:3358). Executed 0events.

Sun Jul8 23:36:01 2018 - [info] End of log messages from 10.0.203.117.

Sun Jul8 23:36:01 2018 - [info] -- Slave on host 10.0.203.117(10.0.203.117:3358) started.

Sun Jul8 23:36:01 2018 - [info] All new slave servers recovered successfully.

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] * Phase 5: New master cleanup phase..

Sun Jul8 23:36:01 2018 - [info]

Sun Jul8 23:36:01 2018 - [info] Resetting slave infoon the new master..

Sun Jul8 23:36:01 2018 - [info] 10.0.203.104: Resetting slave infosucceeded.

Sun Jul8 23:36:01 2018 - [info] Master failover to 10.0.203.104(10.0.203.104:3358) completed successfully.

Sun Jul8 23:36:01 2018 - [info]----- Failover Report -----app1: MySQL Master failover10.0.203.109(10.0.203.109:3358) to 10.0.203.104(10.0.203.104:3358) succeeded

Master10.0.203.109(10.0.203.109:3358) is down!Check MHA Manager logs at localhost.localdomain:/var/log/masterha/app1/manager.log fordetails.

Started automated(non-interactive) failover.

Selected10.0.203.104(10.0.203.104:3358) as a new master.10.0.203.104(10.0.203.104:3358): OK: Applying all logs succeeded.10.0.203.117(10.0.203.117:3358): OK: Slave started, replicating from 10.0.203.104(10.0.203.104:3358)10.0.203.104(10.0.203.104:3358): Resetting slave infosucceeded.

Master failover to10.0.203.104(10.0.203.104:3358) completed successfully.