目录
1、mysql架构
mysql至少一主一从,一般设置为一主两从,且binlog日志目录一定要一致
2、mha软件
一个mha manager管理节点和三个node节点托管mysql实例,管理节点需单独启动,node节点不需要
3、mha配置文件
mha manager配置文件app1.cnf配置 一般存储在/etc/mastermha/下
4、核心脚本
在主库上绑定vip,通过2个脚本管理,一个是故障自动切换的脚本 master_ip_failover,一个是手动切换脚本 master_ip_online_change,vip依赖脚本漂移
5、切换日志
5.1 手动切换
[root@Kary0 bin]# masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=192.168.56.21 --new_master_port=3306 --orig_master_is_new_slave
输出日志:
Tue Jul 12 14:41:37 2022 - [info] MHA::MasterRotate version 0.58.
Tue Jul 12 14:41:37 2022 - [info] Starting online master switch..
Tue Jul 12 14:41:37 2022 - [info]
Tue Jul 12 14:41:37 2022 - [info] * Phase 1: Configuration Check Phase..
Tue Jul 12 14:41:37 2022 - [info]
Tue Jul 12 14:41:37 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Jul 12 14:41:37 2022 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Tue Jul 12 14:41:37 2022 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Tue Jul 12 14:41:37 2022 - [info] GTID failover mode = 0
Tue Jul 12 14:41:37 2022 - [info] Current Alive Master: 192.168.56.20(192.168.56.20:3306)
Tue Jul 12 14:41:37 2022 - [info] Alive Slaves:
Tue Jul 12 14:41:37 2022 - [info] 192.168.56.21(192.168.56.21:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 14:41:37 2022 - [info] Replicating from 192.168.56.20(192.168.56.20:3306)
Tue Jul 12 14:41:37 2022 - [info] 192.168.56.22(192.168.56.22:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 14:41:37 2022 - [info] Replicating from 192.168.56.20(192.168.56.20:3306)It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.56.20(192.168.56.20:3306)? (YES/no): yes
FLUSH NO_WRITE_TO_BINLOG TABLES 关闭所有打开的表,强制关闭所有正在使用的表,不写入binlog,比flush tables方式更柔和
Tue Jul 12 14:41:59 2022 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Tue Jul 12 14:41:59 2022 - [info] ok.
Tue Jul 12 14:41:59 2022 - [info] Checking MHA is not monitoring or doing failover..
Tue Jul 12 14:41:59 2022 - [info] Checking replication health on 192.168.56.21..
Tue Jul 12 14:41:59 2022 - [info] ok.
Tue Jul 12 14:41:59 2022 - [info] Checking replication health on 192.168.56.22..
Tue Jul 12 14:41:59 2022 - [info] ok.
Tue Jul 12 14:41:59 2022 - [info] 192.168.56.21 can be new master.
Tue Jul 12 14:41:59 2022 - [info]
From:
192.168.56.20(192.168.56.20:3306) (current master) 老主库
+--192.168.56.21(192.168.56.21:3306) slave节点
+--192.168.56.22(192.168.56.22:3306) slave节点To:
192.168.56.21(192.168.56.21:3306) (new master) 新主库
+--192.168.56.22(192.168.56.22:3306) slave节点
+--192.168.56.20(192.168.56.20:3306) slave节点Starting master switch from 192.168.56.20(192.168.56.20:3306) to 192.168.56.21(192.168.56.21:3306)? (yes/NO): yes 确认新主库
Tue Jul 12 14:42:13 2022 - [info] Checking whether 192.168.56.21(192.168.56.21:3306) is ok for the new master.. 检查老主库是否正常
Tue Jul 12 14:42:13 2022 - [info] ok.
Tue Jul 12 14:42:13 2022 - [info] 192.168.56.20(192.168.56.20:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host. 检查老主库是否上游有主库
Tue Jul 12 14:42:13 2022 - [info] 192.168.56.20(192.168.56.20:3306): Resetting slave pointing to the dummy host.
Tue Jul 12 14:42:13 2022 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Jul 12 14:42:13 2022 - [info] 配置检查完成
Tue Jul 12 14:42:13 2022 - [info] * Phase 2: Rejecting updates Phase..
Tue Jul 12 14:42:13 2022 - [info]
master_ip_online_change_script is not defined. If you do not disable writes on the current master manually, applications keep writing on the current master. Is it ok to proceed? (yes/NO): yes
Tue Jul 12 14:42:38 2022 - [info] Locking all tables on the orig master to reject updates from everybody (including root):老主库要阻塞写
Tue Jul 12 14:42:38 2022 - [info] Executing FLUSH TABLES WITH READ LOCK..
Tue Jul 12 14:42:38 2022 - [info] ok.
Tue Jul 12 14:42:38 2022 - [info] Orig master binlog:pos is mysql-bin.000003:597.
Tue Jul 12 14:42:38 2022 - [info] Waiting to execute all relay logs on 192.168.56.21(192.168.56.21:3306)..等待所有的中继日志在新主应用完
Tue Jul 12 14:42:38 2022 - [info] master_pos_wait(mysql-bin.000003:597) completed on 192.168.56.21(192.168.56.21:3306). Executed 0 events.
Tue Jul 12 14:42:38 2022 - [info] done.
Tue Jul 12 14:42:38 2022 - [info] Getting new master's binlog name and position..获取新主位点
Tue Jul 12 14:42:38 2022 - [info] mysql-bin.000002:154 位点
Tue Jul 12 14:42:38 2022 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.56.21', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=154, MASTER_USER='root', MASTER_PASSWORD='xxx'; 确认新主位点,重新同步新主
Tue Jul 12 14:42:38 2022 - [info]
Tue Jul 12 14:42:38 2022 - [info] * Switching slaves in parallel..
Tue Jul 12 14:42:38 2022 - [info]
Tue Jul 12 14:42:38 2022 - [info] -- Slave switch on host 192.168.56.22(192.168.56.22:3306) started, pid: 12172
Tue Jul 12 14:42:38 2022 - [info]
Tue Jul 12 14:42:38 2022 - [info] Log messages from 192.168.56.22 ...
Tue Jul 12 14:42:38 2022 - [info]
Tue Jul 12 14:42:38 2022 - [info] Waiting to execute all relay logs on 192.168.56.22(192.168.56.22:3306)..等待应用完中集继日志
Tue Jul 12 14:42:38 2022 - [info] master_pos_wait(mysql-bin.000003:597) completed on 192.168.56.22(192.168.56.22:3306). Executed 0 events.
Tue Jul 12 14:42:38 2022 - [info] done.
Tue Jul 12 14:42:38 2022 - [info] Resetting slave 192.168.56.22(192.168.56.22:3306) and starting replication from the new master 192.168.56.21(192.168.56.21:3306)..
Tue Jul 12 14:42:38 2022 - [info] Executed CHANGE MASTER. 同步到新主
Tue Jul 12 14:42:38 2022 - [info] Slave started.
Tue Jul 12 14:42:38 2022 - [info] End of log messages from 192.168.56.22 ...
Tue Jul 12 14:42:38 2022 - [info]
Tue Jul 12 14:42:38 2022 - [info] -- Slave switch on host 192.168.56.22(192.168.56.22:3306) succeeded.
Tue Jul 12 14:42:38 2022 - [info] Unlocking all tables on the orig master:
Tue Jul 12 14:42:38 2022 - [info] Executing UNLOCK TABLES..老主库解锁
Tue Jul 12 14:42:38 2022 - [info] ok.
Tue Jul 12 14:42:38 2022 - [info] Starting orig master as a new slave..老主降为slave
Tue Jul 12 14:42:38 2022 - [info] Resetting slave 192.168.56.20(192.168.56.20:3306) and starting replication from the new master 192.168.56.21(192.168.56.21:3306)..
Tue Jul 12 14:42:38 2022 - [info] Executed CHANGE MASTER.同步到新主
Tue Jul 12 14:42:38 2022 - [info] Slave started.
Tue Jul 12 14:42:38 2022 - [info] All new slave servers switched successfully. 所有slave切换成功
Tue Jul 12 14:42:38 2022 - [info]
Tue Jul 12 14:42:38 2022 - [info] * Phase 5: New master cleanup phase..
Tue Jul 12 14:42:38 2022 - [info]
Tue Jul 12 14:42:38 2022 - [info] 192.168.56.21: Resetting slave info succeeded.
Tue Jul 12 14:42:38 2022 - [info] Switching master to 192.168.56.21(192.168.56.21:3306) completed successfully.
5.2 故障切换
日志
[root@Kary0 masterha]# tail -f /etc/masterha/app1.log
Tue Jul 12 16:49:40 2022 - [info] Checking master_ip_failover_script status:
Tue Jul 12 16:49:40 2022 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.56.21 --orig_master_ip=192.168.56.21 --orig_master_port=3306
Checking the Status of the script.. OK
Tue Jul 12 16:49:40 2022 - [info] OK.
Tue Jul 12 16:49:40 2022 - [warning] shutdown_script is not defined.
Tue Jul 12 16:49:40 2022 - [info] Set master ping interval 1 seconds.
Tue Jul 12 16:49:40 2022 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Tue Jul 12 16:49:40 2022 - [info] Starting ping health check on 192.168.56.21(192.168.56.21:3306)..
Tue Jul 12 16:49:40 2022 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..开始切换:
Tue Jul 12 16:50:53 2022 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) 发现无法连接
Tue Jul 12 16:50:53 2022 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/usr/local/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin
Tue Jul 12 16:50:54 2022 - [info] HealthCheck: SSH to 192.168.56.21 is reachable.
Tue Jul 12 16:50:54 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.21' (111)) 一次探活
Tue Jul 12 16:50:54 2022 - [warning] Connection failed 2 time(s)..
Tue Jul 12 16:50:55 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.21' (111)) 二次探活
Tue Jul 12 16:50:55 2022 - [warning] Connection failed 3 time(s)..
Tue Jul 12 16:50:56 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.21' (111)) 三次探活
Tue Jul 12 16:50:56 2022 - [warning] Connection failed 4 time(s)..
Tue Jul 12 16:50:56 2022 - [warning] Master is not reachable from health checker!
Tue Jul 12 16:50:56 2022 - [warning] Master 192.168.56.21(192.168.56.21:3306) is not reachable!
Tue Jul 12 16:50:56 2022 - [warning] SSH is reachable.
Tue Jul 12 16:50:56 2022 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..
Tue Jul 12 16:50:56 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Jul 12 16:50:56 2022 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Tue Jul 12 16:50:56 2022 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Tue Jul 12 16:50:56 2022 - [info] GTID failover mode = 0
Tue Jul 12 16:50:56 2022 - [info] Dead Servers:
Tue Jul 12 16:50:56 2022 - [info] 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:56 2022 - [info] Alive Servers:
Tue Jul 12 16:50:56 2022 - [info] 192.168.56.20(192.168.56.20:3306)
Tue Jul 12 16:50:56 2022 - [info] 192.168.56.22(192.168.56.22:3306)
Tue Jul 12 16:50:56 2022 - [info] Alive Slaves:
Tue Jul 12 16:50:56 2022 - [info] 192.168.56.20(192.168.56.20:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:56 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:56 2022 - [info] 192.168.56.22(192.168.56.22:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:56 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:56 2022 - [info] Checking slave configurations..
Tue Jul 12 16:50:56 2022 - [info] read_only=1 is not set on slave 192.168.56.22(192.168.56.22:3306).
Tue Jul 12 16:50:56 2022 - [warning] relay_log_purge=0 is not set on slave 192.168.56.22(192.168.56.22:3306).
Tue Jul 12 16:50:56 2022 - [info] Checking replication filtering settings..
Tue Jul 12 16:50:56 2022 - [info] Replication filtering check ok.
Tue Jul 12 16:50:56 2022 - [info] Master is down!
Tue Jul 12 16:50:56 2022 - [info] Terminating monitoring script.
Tue Jul 12 16:50:56 2022 - [info] Got exit code 20 (Master dead).
Tue Jul 12 16:50:56 2022 - [info] MHA::MasterFailover version 0.58.
Tue Jul 12 16:50:56 2022 - [info] Starting master failover.
Tue Jul 12 16:50:56 2022 - [info]
Tue Jul 12 16:50:56 2022 - [info] * Phase 1: Configuration Check Phase..
Tue Jul 12 16:50:56 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] GTID failover mode = 0
Tue Jul 12 16:50:57 2022 - [info] Dead Servers:
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:57 2022 - [info] Checking master reachability via MySQL(double check)...
Tue Jul 12 16:50:57 2022 - [info] ok.
Tue Jul 12 16:50:57 2022 - [info] Alive Servers:
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.20(192.168.56.20:3306)
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.22(192.168.56.22:3306)
Tue Jul 12 16:50:57 2022 - [info] Alive Slaves:
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.20(192.168.56.20:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:57 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.22(192.168.56.22:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:57 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:57 2022 - [info] Starting Non-GTID based failover.
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Jul 12 16:50:57 2022 - [info] Executing master IP deactivation script:
Tue Jul 12 16:50:57 2022 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.56.21 --orig_master_ip=192.168.56.21 --orig_master_port=3306 --command=stopssh --ssh_user=root
Disabling the VIP on old master: 192.168.56.21
Tue Jul 12 16:50:57 2022 - [info] done.
Tue Jul 12 16:50:57 2022 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Jul 12 16:50:57 2022 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] * Phase 3: Master Recovery Phase..
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] The latest binary log file/position on all slaves is mysql-bin.000003:154
Tue Jul 12 16:50:57 2022 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.20(192.168.56.20:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:57 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.22(192.168.56.22:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:57 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:57 2022 - [info] The oldest binary log file/position on all slaves is mysql-bin.000003:154
Tue Jul 12 16:50:57 2022 - [info] Oldest slaves:
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.20(192.168.56.20:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:57 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:57 2022 - [info] 192.168.56.22(192.168.56.22:3306) Version=5.7.26-log (oldest major version between slaves) log-bin:enabled
Tue Jul 12 16:50:57 2022 - [info] Replicating from 192.168.56.21(192.168.56.21:3306)
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Tue Jul 12 16:50:57 2022 - [info]
Tue Jul 12 16:50:57 2022 - [info] Fetching dead master's binary logs..
Tue Jul 12 16:50:57 2022 - [info] Executing command on the dead master 192.168.56.21(192.168.56.21:3306): save_binary_logs --command=save --start_file=mysql-bin.000003 --start_pos=154 --binlog_dir=/usr/local/mysql/data --output_file=/tmp/saved_master_binlog_from_192.168.56.21_3306_20220712165056.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58
Creating /tmp if not exists.. ok.
Concat binary/relay logs from mysql-bin.000003 pos 154 to mysql-bin.000003 EOF into /tmp/saved_master_binlog_from_192.168.56.21_3306_20220712165056.binlog ..
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 154.. ok.
No need to dump effective binlog data from /usr/local/mysql/data/mysql-bin.000003 (pos starts 154, filesize 154). Skipping.
Binlog Checksum enabled
/tmp/saved_master_binlog_from_192.168.56.21_3306_20220712165056.binlog has no effective data events.
Event not exists.
Tue Jul 12 16:50:58 2022 - [info] Additional events were not found from the orig master. No need to save.
Tue Jul 12 16:50:58 2022 - [info]
Tue Jul 12 16:50:58 2022 - [info] * Phase 3.3: Determining New Master Phase..
Tue Jul 12 16:50:58 2022 - [info]
Tue Jul 12 16:50:58 2022 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Tue Jul 12 16:50:58 2022 - [info] All slaves received relay logs to the same position. No need to resync each other.
Tue Jul 12 16:50:58 2022 - [info] Searching new master from slaves..
Tue Jul 12 16:50:58 2022 - [info] Candidate masters from the configuration file:
Tue Jul 12 16:50:58 2022 - [info] Non-candidate masters:
Tue Jul 12 16:50:58 2022 - [info] New master is 192.168.56.20(192.168.56.20:3306)
Tue Jul 12 16:50:58 2022 - [info] Starting master failover..
Tue Jul 12 16:50:58 2022 - [info]
From:
192.168.56.21(192.168.56.21:3306) (current master)
+--192.168.56.20(192.168.56.20:3306)
+--192.168.56.22(192.168.56.22:3306)To:
192.168.56.20(192.168.56.20:3306) (new master)
+--192.168.56.22(192.168.56.22:3306)
Tue Jul 12 16:50:58 2022 - [info]
Tue Jul 12 16:50:58 2022 - [info] * Phase 3.4: New Master Diff Log Generation Phase..
Tue Jul 12 16:50:58 2022 - [info]
Tue Jul 12 16:50:58 2022 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Jul 12 16:50:58 2022 - [info]
Tue Jul 12 16:50:58 2022 - [info] * Phase 3.5: Master Log Apply Phase..
Tue Jul 12 16:50:58 2022 - [info]
Tue Jul 12 16:50:58 2022 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Jul 12 16:50:58 2022 - [info] Starting recovery on 192.168.56.20(192.168.56.20:3306)..
Tue Jul 12 16:50:58 2022 - [info] This server has all relay logs. Waiting all logs to be applied..
Tue Jul 12 16:50:58 2022 - [info] done.
Tue Jul 12 16:50:58 2022 - [info] All relay logs were successfully applied.
Tue Jul 12 16:50:58 2022 - [info] Getting new master's binlog name and position..
Tue Jul 12 16:50:58 2022 - [info] mysql-bin.000003:597
Tue Jul 12 16:50:58 2022 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.56.20', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=597, MASTER_USER='root', MASTER_PASSWORD='xxx';
Tue Jul 12 16:50:58 2022 - [info] Executing master IP activate script:
Tue Jul 12 16:50:58 2022 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.56.21 --orig_master_ip=192.168.56.21 --orig_master_port=3306 --new_master_host=192.168.56.20 --new_master_ip=192.168.56.20 --new_master_port=3306 --new_master_user='root' --new_master_password=xxx
Unknown option: new_master_user
Unknown option: new_master_password
Enabling the VIP - 192.168.100.100 on the new master - 192.168.56.20
Tue Jul 12 16:51:02 2022 - [info] OK.
Tue Jul 12 16:51:02 2022 - [info] Setting read_only=0 on 192.168.56.20(192.168.56.20:3306)..
Tue Jul 12 16:51:02 2022 - [info] ok.
Tue Jul 12 16:51:02 2022 - [info] ** Finished master recovery successfully.
Tue Jul 12 16:51:02 2022 - [info] * Phase 3: Master Recovery Phase completed.
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] * Phase 4: Slaves Recovery Phase..
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] -- Slave diff file generation on host 192.168.56.22(192.168.56.22:3306) started, pid: 20891. Check tmp log /etc/masterha//192.168.56.22_3306_20220712165056.log if it takes time..
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] Log messages from 192.168.56.22 ...
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Jul 12 16:51:02 2022 - [info] End of log messages from 192.168.56.22.
Tue Jul 12 16:51:02 2022 - [info] -- 192.168.56.22(192.168.56.22:3306) has the latest relay log events.
Tue Jul 12 16:51:02 2022 - [info] Generating relay diff files from the latest slave succeeded.
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] -- Slave recovery on host 192.168.56.22(192.168.56.22:3306) started, pid: 20893. Check tmp log /etc/masterha//192.168.56.22_3306_20220712165056.log if it takes time..
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] Log messages from 192.168.56.22 ...
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] Starting recovery on 192.168.56.22(192.168.56.22:3306)..
Tue Jul 12 16:51:02 2022 - [info] This server has all relay logs. Waiting all logs to be applied..
Tue Jul 12 16:51:02 2022 - [info] done.
Tue Jul 12 16:51:02 2022 - [info] All relay logs were successfully applied.
Tue Jul 12 16:51:02 2022 - [info] Resetting slave 192.168.56.22(192.168.56.22:3306) and starting replication from the new master 192.168.56.20(192.168.56.20:3306)..
Tue Jul 12 16:51:02 2022 - [info] Executed CHANGE MASTER.
Tue Jul 12 16:51:02 2022 - [info] Slave started.
Tue Jul 12 16:51:02 2022 - [info] End of log messages from 192.168.56.22.
Tue Jul 12 16:51:02 2022 - [info] -- Slave recovery on host 192.168.56.22(192.168.56.22:3306) succeeded.
Tue Jul 12 16:51:02 2022 - [info] All new slave servers recovered successfully.
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] * Phase 5: New master cleanup phase..
Tue Jul 12 16:51:02 2022 - [info]
Tue Jul 12 16:51:02 2022 - [info] Resetting slave info on the new master..
Tue Jul 12 16:51:02 2022 - [info] 192.168.56.20: Resetting slave info succeeded.
Tue Jul 12 16:51:02 2022 - [info] Master failover to 192.168.56.20(192.168.56.20:3306) completed successfully.
Tue Jul 12 16:51:02 2022 - [info] Deleted server02 entry from /etc/masterha/app1.cnf .
Tue Jul 12 16:51:02 2022 - [info]----- Failover Report -----
app1: MySQL Master failover 192.168.56.21(192.168.56.21:3306) to 192.168.56.20(192.168.56.20:3306) succeeded
Master 192.168.56.21(192.168.56.21:3306) is down!
Check MHA Manager logs at Kary0.com:/etc/masterha/app1.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.56.21(192.168.56.21:3306)
The latest slave 192.168.56.20(192.168.56.20:3306) has all relay logs for recovery.
Selected 192.168.56.20(192.168.56.20:3306) as a new master.
192.168.56.20(192.168.56.20:3306): OK: Applying all logs succeeded.
192.168.56.20(192.168.56.20:3306): OK: Activated master IP address.
192.168.56.22(192.168.56.22:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.56.22(192.168.56.22:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.56.20(192.168.56.20:3306)
192.168.56.20(192.168.56.20:3306): Resetting slave info succeeded.
Master failover to 192.168.56.20(192.168.56.20:3306) completed successfully.