MHA切换日志

测试 一主两从 

mysql5.7 GTID 

1 启动MHA进程

sh start.sh op-yanhaihang-mha-mysql

2 制造流量

mysqlslap -udbadmin_01 -p*** -hdx-op-yanhaihang-mha-mysql1-online -P 3306 --auto-generate-sql --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --concurrency=2 --number-of-queries=10000000

3 制造延迟 备主延迟大 ，另外一个从库 延迟小一点

先停止备主同步进程

stop slave;

再停止另外一个从库 复制进程

stop slave;

4 停止主库进程 手动kill

先 kill mysqld_safe

再kill mysqld

否则MySQL会再启动

切换日志 

Thu Jun 24 16:27:11 2021 - [warning] Got error on MySQL connect ping: DBI connect(';host=10.11.42.110;port=3306;mysql_connect_timeout=1','root',...) failed: Can't connect to MySQL server on '10.11.42.110' (111) at /usr/share/perl5/vendor_perl/MHA/HealthCheck.pm line 97.
2003 (Can't connect to MySQL server on '10.11.42.110' (111))
Thu Jun 24 16:27:11 2021 - [info] Executing secondary network check script: masterha_secondary_check -s dx-op-yanhaihang-mha-mysql1-online  -s  lf-op-yanhaihang-mha-mysql2-online  --user=maintain  --master_host=dx-op-yanhaihang-mha-mysql1-online  --master_ip=10.11.42.110  --master_port=3306 --master_user=root --master_password=9WJKXw8e --ping_type=CONNECT
Thu Jun 24 16:27:11 2021 - [info] Executing SSH check script: exit 0
Thu Jun 24 16:27:12 2021 - [info] HealthCheck: SSH to dx-op-yanhaihang-mha-mysql1-online is reachable.
Monitoring server dx-op-yanhaihang-mha-mysql1-online is reachable, Master is not reachable from dx-op-yanhaihang-mha-mysql1-online. OK.
Monitoring server lf-op-yanhaihang-mha-mysql2-online is reachable, Master is not reachable from lf-op-yanhaihang-mha-mysql2-online. OK.
Thu Jun 24 16:27:12 2021 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Thu Jun 24 16:27:14 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.11.42.110' (111))
Thu Jun 24 16:27:14 2021 - [warning] Connection failed 2 time(s)..
Thu Jun 24 16:27:17 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.11.42.110' (111))
Thu Jun 24 16:27:17 2021 - [warning] Connection failed 3 time(s)..
Thu Jun 24 16:27:20 2021 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.11.42.110' (111))
Thu Jun 24 16:27:20 2021 - [warning] Connection failed 4 time(s)..
Thu Jun 24 16:27:20 2021 - [warning] Master is not reachable from health checker!
Thu Jun 24 16:27:20 2021 - [warning] Master dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306) is not reachable!
Thu Jun 24 16:27:20 2021 - [warning] SSH is reachable.
Thu Jun 24 16:27:20 2021 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /home/maintain/masterha/etc/op-yanhaihang-mha-mysql.conf again, and trying to connect to all servers to check server status..
Thu Jun 24 16:27:20 2021 - [info] Reading default configuration from /etc/masterha_default.cnf..
Thu Jun 24 16:27:20 2021 - [info] Reading application default configuration from /home/maintain/masterha/etc/op-yanhaihang-mha-mysql.conf..
Thu Jun 24 16:27:20 2021 - [info] Reading server configuration from /home/maintain/masterha/etc/op-yanhaihang-mha-mysql.conf..
Thu Jun 24 16:27:21 2021 - [warning] SQL Thread is stopped(no error) on lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)
Thu Jun 24 16:27:21 2021 - [warning] SQL Thread is stopped(no error) on lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)
Thu Jun 24 16:27:21 2021 - [info] GTID failover mode = 1
Thu Jun 24 16:27:21 2021 - [info] Dead Servers:
Thu Jun 24 16:27:21 2021 - [info]   dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:21 2021 - [info] Alive Servers:
Thu Jun 24 16:27:21 2021 - [info]   lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)
Thu Jun 24 16:27:21 2021 - [info]   lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)
Thu Jun 24 16:27:21 2021 - [info] Alive Slaves:
Thu Jun 24 16:27:21 2021 - [info]   lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)  Version=5.7.29-log (oldest major version between slaves) log-bin:enabled
Thu Jun 24 16:27:21 2021 - [info]     GTID ON
Thu Jun 24 16:27:21 2021 - [info]     Replicating from dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:21 2021 - [info]   lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)  Version=5.7.29-log (oldest major version between slaves) log-bin:enabled
Thu Jun 24 16:27:21 2021 - [info]     GTID ON
Thu Jun 24 16:27:21 2021 - [info]     Replicating from dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:21 2021 - [info]     Not candidate for the new Master (no_master is set)
Thu Jun 24 16:27:21 2021 - [info] Checking slave configurations..
Thu Jun 24 16:27:21 2021 - [info]  read_only=1 is not set on slave lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306).
Thu Jun 24 16:27:21 2021 - [info]  read_only=1 is not set on slave lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306).
Thu Jun 24 16:27:21 2021 - [info] Checking replication filtering settings..
Thu Jun 24 16:27:21 2021 - [info]  Replication filtering check ok.
Thu Jun 24 16:27:21 2021 - [info] Master is down!
Thu Jun 24 16:27:21 2021 - [info] Terminating monitoring script.
Thu Jun 24 16:27:21 2021 - [info] Got exit code 20 (Master dead).
Thu Jun 24 16:27:21 2021 - [info] MHA::MasterFailover version 0.56.
Thu Jun 24 16:27:21 2021 - [info] Starting master failover.
Thu Jun 24 16:27:21 2021 - [info]
Thu Jun 24 16:27:21 2021 - [info] * Phase 1: Configuration Check Phase..
Thu Jun 24 16:27:21 2021 - [info]
Thu Jun 24 16:27:22 2021 - [info] HealthCheck: SSH to dx-op-yanhaihang-mha-mysql1-online is reachable.
Thu Jun 24 16:27:22 2021 - [info] Binlog server dx-op-yanhaihang-mha-mysql1-online is reachable.
Thu Jun 24 16:27:23 2021 - [warning] SQL Thread is stopped(no error) on lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)
Thu Jun 24 16:27:23 2021 - [warning] SQL Thread is stopped(no error) on lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)
Thu Jun 24 16:27:23 2021 - [info] GTID failover mode = 1
Thu Jun 24 16:27:23 2021 - [info] Dead Servers:
Thu Jun 24 16:27:23 2021 - [info]   dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:23 2021 - [info] Checking master reachability via MySQL(double check)...
Thu Jun 24 16:27:23 2021 - [info]  ok.
Thu Jun 24 16:27:23 2021 - [info] Alive Servers:
Thu Jun 24 16:27:23 2021 - [info]   lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)
Thu Jun 24 16:27:23 2021 - [info]   lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)
Thu Jun 24 16:27:23 2021 - [info] Alive Slaves:
Thu Jun 24 16:27:23 2021 - [info]   lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)  Version=5.7.29-log (oldest major version between slaves) log-bin:enabled
Thu Jun 24 16:27:23 2021 - [info]     GTID ON
Thu Jun 24 16:27:23 2021 - [info]     Replicating from dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:23 2021 - [info]   lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)  Version=5.7.29-log (oldest major version between slaves) log-bin:enabled
Thu Jun 24 16:27:23 2021 - [info]     GTID ON
Thu Jun 24 16:27:23 2021 - [info]     Replicating from dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:23 2021 - [info]     Not candidate for the new Master (no_master is set)
Thu Jun 24 16:27:23 2021 - [info]  Starting SQL thread on lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306) ..
Thu Jun 24 16:27:23 2021 - [info]   done.
Thu Jun 24 16:27:23 2021 - [info]  Starting SQL thread on lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306) ..
Thu Jun 24 16:27:23 2021 - [info]   done.
Thu Jun 24 16:27:23 2021 - [info] Starting GTID based failover.
Thu Jun 24 16:27:23 2021 - [info]
Thu Jun 24 16:27:23 2021 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Jun 24 16:27:23 2021 - [info]
Thu Jun 24 16:27:23 2021 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Jun 24 16:27:23 2021 - [info]
Thu Jun 24 16:27:23 2021 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Jun 24 16:27:23 2021 - [info] Executing master IP deactivation script:
Thu Jun 24 16:27:23 2021 - [info]   /home/maintain/masterha/script/master_ip_failover --orig_master_host=dx-op-yanhaihang-mha-mysql1-online --orig_master_ip=10.11.42.110 --orig_master_port=3306 --command=stopssh --ssh_user=maintain
will kill mysql processes
pids:['739675\n']
ROUND 1
kill mysql process,pids=['739675\n']
kill_cmd:ssh maintain@dx-op-yanhaihang-mha-mysql1-online "kill -9 739675
"
pids:['754089\n']
ROUND 2
kill mysql process,pids=['754089\n']
kill_cmd:ssh maintain@dx-op-yanhaihang-mha-mysql1-online "kill -9 754089
"
pids:['754717\n']
ROUND 3
kill mysql process,pids=['754717\n']
kill_cmd:ssh maintain@dx-op-yanhaihang-mha-mysql1-online "kill -9 754717
"
Thu Jun 24 16:27:29 2021 - [info]  done.
Thu Jun 24 16:27:29 2021 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Jun 24 16:27:29 2021 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Thu Jun 24 16:27:29 2021 - [info]
Thu Jun 24 16:27:29 2021 - [info] * Phase 3: Master Recovery Phase..
Thu Jun 24 16:27:29 2021 - [info]
Thu Jun 24 16:27:29 2021 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Jun 24 16:27:29 2021 - [info]
Thu Jun 24 16:27:29 2021 - [info] The latest binary log file/position on all slaves is mysql-bin.000013:52687819
Thu Jun 24 16:27:29 2021 - [info] Retrieved Gtid Set: 916d122d-ce9c-11eb-bd31-aabb0a0b2a6e:5003813-5490801
Thu Jun 24 16:27:29 2021 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Jun 24 16:27:29 2021 - [info]   lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)  Version=5.7.29-log (oldest major version between slaves) log-bin:enabled
Thu Jun 24 16:27:29 2021 - [info]     GTID ON
Thu Jun 24 16:27:29 2021 - [info]     Replicating from dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:29 2021 - [info]     Not candidate for the new Master (no_master is set)
Thu Jun 24 16:27:29 2021 - [info] The oldest binary log file/position on all slaves is mysql-bin.000013:25970934
Thu Jun 24 16:27:29 2021 - [info] Retrieved Gtid Set: 916d122d-ce9c-11eb-bd31-aabb0a0b2a6e:5263079-5424506
Thu Jun 24 16:27:29 2021 - [info] Oldest slaves:
Thu Jun 24 16:27:29 2021 - [info]   lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)  Version=5.7.29-log (oldest major version between slaves) log-bin:enabled
Thu Jun 24 16:27:29 2021 - [info]     GTID ON
Thu Jun 24 16:27:29 2021 - [info]     Replicating from dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:29 2021 - [info]
Thu Jun 24 16:27:29 2021 - [info] * Phase 3.3: Determining New Master Phase..
Thu Jun 24 16:27:29 2021 - [info]
Thu Jun 24 16:27:29 2021 - [info] Searching new master from slaves..
Thu Jun 24 16:27:29 2021 - [info]  Candidate masters from the configuration file:
Thu Jun 24 16:27:29 2021 - [info]  Non-candidate masters:
Thu Jun 24 16:27:29 2021 - [info]   lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)  Version=5.7.29-log (oldest major version between slaves) log-bin:enabled
Thu Jun 24 16:27:29 2021 - [info]     GTID ON
Thu Jun 24 16:27:29 2021 - [info]     Replicating from dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Thu Jun 24 16:27:29 2021 - [info]     Not candidate for the new Master (no_master is set)
Thu Jun 24 16:27:29 2021 - [info]  Searching from all slaves which have received the latest relay log events..
Thu Jun 24 16:27:29 2021 - [info]   Not found.
Thu Jun 24 16:27:29 2021 - [info]  Searching from all slaves..
Thu Jun 24 16:27:29 2021 - [info] New master is lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)
Thu Jun 24 16:27:29 2021 - [info] Starting master failover..
Thu Jun 24 16:27:29 2021 - [info]
From:
dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306) (current master)
 +--lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)
 +--lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)

To:
lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306) (new master)
 +--lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)
Thu Jun 24 16:27:29 2021 - [info]
Thu Jun 24 16:27:29 2021 - [info] * Phase 3.3: New Master Recovery Phase..
Thu Jun 24 16:27:29 2021 - [info]
Thu Jun 24 16:27:29 2021 - [info]  Waiting all logs to be applied..
Thu Jun 24 16:27:29 2021 - [info]   done.
Thu Jun 24 16:27:29 2021 - [info]  Replicating from the latest slave lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306) and waiting to apply..
Thu Jun 24 16:27:29 2021 - [info]  Waiting all logs to be applied on the latest slave..
Thu Jun 24 16:27:29 2021 - [info]  Resetting slave lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306) and starting replication from the new master lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306)..
Thu Jun 24 16:27:29 2021 - [info]  Executed CHANGE MASTER.
Thu Jun 24 16:27:30 2021 - [info]  Slave started.
Thu Jun 24 16:27:30 2021 - [info]  Waiting to execute all relay logs on lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)..
Thu Jun 24 16:27:40 2021 - [info]  master_pos_wait(mysql-bin.000003:403936248) completed on lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306). Executed 130 events.
Thu Jun 24 16:27:40 2021 - [info]   done.
Thu Jun 24 16:27:40 2021 - [info]   done.
Thu Jun 24 16:27:40 2021 - [info] -- Saving binlog from host dx-op-yanhaihang-mha-mysql1-online started, pid: 500797
Thu Jun 24 16:27:43 2021 - [info]
Thu Jun 24 16:27:43 2021 - [info] Log messages from dx-op-yanhaihang-mha-mysql1-online ...
Thu Jun 24 16:27:43 2021 - [info]
Thu Jun 24 16:27:40 2021 - [info] Fetching binary logs from binlog server dx-op-yanhaihang-mha-mysql1-online..
Thu Jun 24 16:27:40 2021 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=mysql-bin.000013  --start_pos=52687819 --output_file=/home/shared/masterha/saved_binlog_binlog1_20210624162721.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.7.29-log  --binlog_dir=/home/maintain/mysqlbinlog
  Creating /home/shared/masterha if not exists..    ok.
 Concat binary/relay logs from mysql-bin.000013 pos 52687819 to mysql-bin.000013 EOF into /home/shared/masterha/saved_binlog_binlog1_20210624162721.binlog ..
 Concat succeeded.
Thu Jun 24 16:27:42 2021 - [info] scp from maintain@10.11.42.110:/home/shared/masterha/saved_binlog_binlog1_20210624162721.binlog to local:/home/maintain/masterha/op-yanhaihang-mha-mysql/saved_binlog_dx-op-yanhaihang-mha-mysql1-online_binlog1_20210624162721.binlog succeeded.
Thu Jun 24 16:27:43 2021 - [info] End of log messages from dx-op-yanhaihang-mha-mysql1-online.
Thu Jun 24 16:27:43 2021 - [info] Saved mysqlbinlog size from dx-op-yanhaihang-mha-mysql1-online is 36379044 bytes.
Thu Jun 24 16:27:43 2021 - [info] Applying differential binlog /home/maintain/masterha/op-yanhaihang-mha-mysql/saved_binlog_dx-op-yanhaihang-mha-mysql1-online_binlog1_20210624162721.binlog ..
Thu Jun 24 16:28:19 2021 - [info] Differential log apply from binlog server succeeded.
Thu Jun 24 16:28:19 2021 - [info] Getting new master's binlog name and position..
Thu Jun 24 16:28:19 2021 - [info]  mysql-bin.000004:201814509
Thu Jun 24 16:28:19 2021 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='lf-op-yanhaihang-mha-mysql1-online or 10.9.34.71', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Thu Jun 24 16:28:19 2021 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000004, 201814509, 19a6d8c3-ce9d-11eb-859c-aabb0a092247:1-1068846,
916d122d-ce9c-11eb-bd31-aabb0a0b2a6e:1-5521422,
be32273a-ce9e-11eb-a839-aabb0a092249:1
Thu Jun 24 16:28:19 2021 - [info] Executing master IP activate script:
Thu Jun 24 16:28:19 2021 - [info]   /home/maintain/masterha/script/master_ip_failover --command=start --ssh_user=maintain --orig_master_host=dx-op-yanhaihang-mha-mysql1-online --orig_master_ip=10.11.42.110 --orig_master_port=3306 --new_master_host=lf-op-yanhaihang-mha-mysql1-online --new_master_ip=10.9.34.71 --new_master_port=3306 --new_master_user='root' --new_master_password='9WJKXw8e'
offline host:dx-op-yanhaihang-mha-mysql1-online
{
  "msg": "Sucess",
  "status": 0
}
update cmdb set dx-op-yanhaihang-mha-mysql1-online is problem
update cmdb set lf-op-yanhaihang-mha-mysql1-online is new master
update cmdb set lf-op-yanhaihang-mha-mysql1-online new roles
{"message": "get dbconfig faild:Traceback (most recent call last):\n  File \"/home/shared/db-dba-service/app/dbfdc/db_config.py\", line 161, in sync_dbconfig\n    e_endpoint = db_endpoint(db,db_name,cluster_id,user_dict)\n  File \"/home/shared/db-dba-service/app/dbfdc/db_config.py\", line 67, in db_endpoint\n    raise Exception(msg)\nException: cluster_id:888870 not find any instance in database_isntance table \n", "success": false}
Thu Jun 24 16:28:20 2021 - [info]  OK.
Thu Jun 24 16:28:20 2021 - [info] ** Finished master recovery successfully.
Thu Jun 24 16:28:20 2021 - [info] * Phase 3: Master Recovery Phase completed.
Thu Jun 24 16:28:20 2021 - [info]
Thu Jun 24 16:28:20 2021 - [info] * Phase 4: Slaves Recovery Phase..
Thu Jun 24 16:28:20 2021 - [info]
Thu Jun 24 16:28:20 2021 - [info]
Thu Jun 24 16:28:20 2021 - [info] * Phase 4.1: Starting Slaves in parallel..
Thu Jun 24 16:28:20 2021 - [info]
Thu Jun 24 16:28:20 2021 - [info] -- Slave recovery on host lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306) started, pid: 501064. Check tmp log /home/maintain/masterha/op-yanhaihang-mha-mysql/lf-op-yanhaihang-mha-mysql2-online_3306_20210624162721.log if it takes time..
Thu Jun 24 16:28:27 2021 - [info]
Thu Jun 24 16:28:27 2021 - [info] Log messages from lf-op-yanhaihang-mha-mysql2-online ...
Thu Jun 24 16:28:27 2021 - [info]
Thu Jun 24 16:28:20 2021 - [info]  Resetting slave lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306) and starting replication from the new master lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)..
Thu Jun 24 16:28:20 2021 - [info]  Executed CHANGE MASTER.
Thu Jun 24 16:28:21 2021 - [info]  Slave started.
Thu Jun 24 16:28:27 2021 - [info]  gtid_wait(19a6d8c3-ce9d-11eb-859c-aabb0a092247:1-1068846,
916d122d-ce9c-11eb-bd31-aabb0a0b2a6e:1-5521422,
be32273a-ce9e-11eb-a839-aabb0a092249:1) completed on lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306). Executed 61 events.
Thu Jun 24 16:28:27 2021 - [info] End of log messages from lf-op-yanhaihang-mha-mysql2-online.
Thu Jun 24 16:28:27 2021 - [info] -- Slave on host lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306) started.
Thu Jun 24 16:28:27 2021 - [info] All new slave servers recovered successfully.
Thu Jun 24 16:28:27 2021 - [info]
Thu Jun 24 16:28:27 2021 - [info] * Phase 5: New master cleanup phase..
Thu Jun 24 16:28:27 2021 - [info]
Thu Jun 24 16:28:27 2021 - [info] Resetting slave info on the new master..
Thu Jun 24 16:28:27 2021 - [info]  lf-op-yanhaihang-mha-mysql1-online: Resetting slave info succeeded.
Thu Jun 24 16:28:27 2021 - [info] Master failover to lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306) completed successfully.
Thu Jun 24 16:28:27 2021 - [info]

----- Failover Report -----

op-yanhaihang-mha-mysql: MySQL Master failover dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306) to lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306) succeeded

Master dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306) is down!

Check MHA Manager logs at lf-op-mha-secondary1-online:/home/maintain/masterha/op-yanhaihang-mha-mysql/manager.log for details.

Started automated(non-interactive) failover.
Invalidated master IP address on dx-op-yanhaihang-mha-mysql1-online(10.11.42.110:3306)
Selected lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306) as a new master.
lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306): OK: Applying all logs succeeded.
lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306): OK: Activated master IP address.
lf-op-yanhaihang-mha-mysql2-online(10.9.34.73:3306): OK: Slave started, replicating from lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306)
lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306): Resetting slave info succeeded.
Master failover to lf-op-yanhaihang-mha-mysql1-online(10.9.34.71:3306) completed successfully.
Thu Jun 24 16:28:27 2021 - [info] Sending mail..