一，基础环境
sql版本：mysql 5.7.17
系统：Centos7.2
VIP： 10.180.160.118
角色 IP地址 serverID 权限
master 10.180.160.113 1 读写
slave1(备master) 10.180.160.113 2 读
slave2 10.180.160.113 3 读
二,安装mysql,配置主从
2.1 安装mysql（略）
直接用ansible推下去，如果是拷贝的data目录，记得修改auto.cnf中UUID，避免三台一样。
2.2 配置主从
mysql配置文件
vi /etc/my.cnf
#MHA MYSQL集群master配置文件
[client]
user=root
socket = /apps/mysql/data/mysql.sock
[mysqld]
socket=/apps/mysql/data/mysql.sock
basedir=/apps/mysql
datadir=/apps/mysql/data
log-error=/apps/mysql/data/mysql_error.log
pid-file=/apps/mysql/data/mysql.pid
#每个server上不一致，见规划
server-id = 1
#read-only=1 #不建设在配置文件中限定只读，但是要记得在slave上限制只读
#mysql5.6已上的特性，开启gtid，必须主从全开
gtid_mode = on
enforce_gtid_consistency = 1
log_slave_updates = 1
#开启半同步复制 否则自动切换主从的时候会报主键错误
plugin_load = "rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so"
loose_rpl_semi_sync_master_enabled = 1
loose_rpl_semi_sync_slave_enabled = 1
loose_rpl_semi_sync_master_timeout = 5000
log-bin=mysql-bin
relay-log = mysql-relay-bin
replicate-wild-ignore-table=mysql.%
replicate-wild-ignore-table=test.%
replicate-wild-ignore-table=information_schema.%
#relay_log_purge=0 #禁止自动删除中继日志(slave配置文件多了下面这两条），如果是MHA开启此项
启动mysql服务
systemctl restart mysqld.service #重启服务
在slave1和slave2上执行
mysql -uroot -pdbaadmin@123 -e "set global read_only=1
从库只读，不建议写在配置文件中
2.3 设置账户
master（10.180.160.113） 上操作
设置root账户
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhsot' IDENTIFIED BY 'dbaadmin@123';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'dabadmin@123';
创建复制账户，并授权可以访问的主机:
mysql> grant replication slave on *.* to 'repl'@'10.180.160.114%' identified by 'repl@123';
mysql> grant replication slave on *.* to 'repl'@'10.180.160.115%' identified by 'repl@123';
mysql> FLUSH PRIVILEGES;
slave1(10.180.160.114)上操作
设置root账户
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhsot' IDENTIFIED BY 'dbaadmin@123';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'dabadmin@123';
创建复制账户，并授权可以访问的主机:
mysql> grant replication slave on *.* to 'repl'@'10.180.160.113%' identified by 'repl@123';
mysql> grant replication slave on *.* to 'repl'@'10.180.160.115%' identified by 'repl@123';
mysql> FLUSH PRIVILEGES;
```
**slave2（10.180.160.115）上操作**
设置root账户
```
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhsot' IDENTIFIED BY 'dbaadmin@123';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'dabadmin@123';
```
创建复制账户，并授权可以访问的主机:
```
mysql> grant replication slave on *.* to 'repl'@'10.180.160.113%' identified by 'repl@123';
mysql> grant replication slave on *.* to 'repl'@'10.180.160.114%' identified by 'repl@123';
mysql> FLUSH PRIVILEGES;
```
2.4 实现主从同步
2.4.1 在mster(10.180.160.113)上备份一份完整的数据：
```
mysqldump --master-data=2 --single-transaction -R --triggers -A > /data/baksql/all.sql;
```
说明：
--master-data=2代表备份时刻记录master的Binlog位置和Position
--single-transaction意思是获取一致性快照
-R意思是备份存储过程和函数
--triggres的意思是备份触发器
-A代表备份所有的库
查看更多信息mysqldump --help
将备份文件拷贝到slave1,slave2上
查看Master主库备份时的binlog名称和位置，MASTER_LOG_FILE和MASTER_LOG_POS：
```
head -n 30 /data/baksql/all.sql | grep 'CHANGE MASTER TO'
-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=154;
```
2.4.2 启动slave1,slave2上mysql服务后，执行change master语句，实现主从同步。
mysql -uroot -p密码 -hlocalhost database < /tmp/备份数据库名.sql; # 例如 all.sql
mysql>stop slave; # 暂停从机复制。
mysql>change master to master_host='10.180.160.113',master_user='repl',master_password='repl@123',master_log_file='mysql-bin.000004',master_log_pos=154; # 配置主从关系。并且log_file和log_pos参数与主库master status状态显示一致。
开启从机复制
```
mysql>start slave;
```
查看状态
```
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.180.160.113
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 154
Relay_Log_File: mysql-relay-bin.000002
Relay_Log_Pos: 320
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,test.%,information_schema.%
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 527
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_UUID: 2ccd6e03-2d96-11e8-9cfc-0050569666a2
Master_Info_File: /apps/mysql/data/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
```
查看master状态
```
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 | 154 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)
```
2.4.3、验证复制是否正常！
############主库10.180.160.113 Master插入一行数据，并查看Position号
创建库
```
mysql> CREATE DATABASE vava;
mysql> use vava;
mysql> CREATE TABLE employee(id int(10), name char(20),phone int(12));
mysql> CREATE TABLE department(dpt_name CHAR(20), dpt_phone INT(12));
mysql> INSERT INTO employee(id,name,phone) VALUES(01, 'Tom' , 110110110);
mysql> INSERT INTO employee VALUES(02, 'Jack' , 119119119);
mysql> INSERT INTO employee (id,name) VALUES(03, 'Rose');
```
对比Position 是否一致。
master上
```
mysql> show master status \G;
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 1526
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: 2ccd6e03-2d96-11e8-9cfc-0050569666a2:1-6
1 row in set (0.00 sec)
ERROR:
No query specified
```
slave1上
```
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.180.160.113
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 1526
Relay_Log_File: mysql-relay-bin.000002
```
slave2上
```
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.180.160.113
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 1526
Relay_Log_File: mysql-relay-bin.000002
```
2.5 环境检查
确保所有主从数据库都开启二进制日志
```
mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin | ON |
+---------------+-------+
1 row in set (0.00 sec)
```
查看从节点，是否设置为只读
```
mysql> show variables like 'read_only';
mysql> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only | ON |
+---------------+-------+
1 row in set (0.00 sec)
```
------------------------------------------------------------------------------------------
**三，配置MHA集群**
3.1. 配置三台机器的ssh互信（三台都要操作）
```
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.180.160.113
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.180.160.114
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.180.160.115
```
#测试是否成功
```
ssh 10.180.160.114 date
```
3.2. 安装MHA软件（在三个节点上都装mha的node软件）
#先安装依赖
```
wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager -y
```
下载软件（方式任选其一）
```
wget https://qiniu.wsfnk.com/mha4mysql-node-0.58-0.el7.centos.noarch.rpm
#wget https://github.com/yoshinorim/mha4mysql-node/releases/download/v0.58/mha4mysql-node-0.58-0.el7.centos.noarch.rpm
rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
```
3.3 仅在manager节点上安装mha管理软件
```
wget https://qiniu.wsfnk.com/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
#wget https://github.com/yoshinorim/mha4mysql-manager/releases/download/v0.58/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
rpm -ivh mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
```
安装mailx用来发送邮件
```
yum install mailx -y
```
3.4 配置MHA（在manager节点上操作）
#创建目录
```
mkdir -p /etc/mha/scripts
```
3.4.1 配置全局配置文件
vim /etc/masterha_default.cnf #一定要是这个路径，不然后期masterha_check_ssh会提示未找到全局文件
```
[server default]
user=root
password=dbaadmin@123
ssh_user=root
repl_user=repl
repl_password=repl@123
ping_interval=1
#master_binlog_dir= /var/lib/mysql,/var/log/mysql
secondary_check_script=masterha_secondary_check -s 10.180.160.113 -s 10.180.160.114 -s 10.180.160.115
master_ip_failover_script="/etc/mha/scripts/master_ip_failover"
master_ip_online_change_script="/etc/mha/scripts/master_ip_online_change"
report_script="/etc/mha/scripts/send_report"
```
3.4.2配置主配置文件
```
vim /etc/mha/app1.cnf
[server default]
manager_workdir=/var/log/mha/app1
manager_log=/var/log/mha/app1/manager.log
[server1]
hostname=10.180.160.113
candidate_master=1
master_binlog_dir="/apps/mysql/binlogs/"
#查看方式 find / -name mysql-bin*
[server2]
hostname=10.180.160.114
candidate_master=1
master_binlog_dir="/apps/mysql/binlogs/"
[server3]
hostname=10.180.160.115
master_binlog_dir="/apps/mysql/binlogs/"
#表示没有机会成为master
no_master=1
```
3.4.3配置VIP
#为了防止脑裂发生,推荐生产环境采用脚本的方式来管理虚拟 ip,而不是使用 keepalived来完成。
```
vim /etc/mha/scripts/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host,
$orig_master_ip,$orig_master_port, $new_master_host, $new_master_ip,$new_master_port
);
#定义VIP变量
my $vip = '10.180.160.118/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens160:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens160:$key down";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub stop_vip() {
return 0 unless ($ssh_user);
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
```
3.4.4.配置报警邮件脚本
#mail邮件发送程序，需要先配置好发送这信息
```
vim /etc/mail.rc
set from=csctbb@163.com
set smtp=smtp.163.com
set smtp-auth-user=csctbb
#拿163邮箱来说这个不是密码，而是授权码
set smtp-auth-password=wqwqwq123
set smtp-auth=login
```
#这是具体的邮件发送脚本
```
vim /etc/mha/scripts/send_report
#!/bin/bash
source /root/.bash_profile
#解析变量
orig_master_host=`echo "$1" | awk -F = '{print $2}'`
new_master_host=`echo "$2" | awk -F = '{print $2}'`
new_slave_hosts=`echo "$3" | awk -F = '{print $2}'`
subject=`echo "$4" | awk -F = '{print $2}'`
body=`echo "$5" | awk -F = '{print $2}'`
#定义收件人地址
email="sjcctbb@163.com"
tac /var/log/mha/app1/manager.log | sed -n 2p | grep 'successfully' > /dev/null
if [ $? -eq 0 ]
then
messages=`echo -e "MHA $subject 主从切换成功\n master:$orig_master_host --> $new_master_host \n $body \n 当前从库:$new_slave_hosts"`
echo "$messages" | mail -s "Mysql 实例宕掉，MHA $subject 切换成功" $email >>/tmp/mailx.log 2>&1
else
messages=`echo -e "MHA $subject 主从切换失败\n master:$orig_master_host --> $new_master_host \n $body" `
echo "$messages" | mail -s ""Mysql 实例宕掉，MHA $subject 切换失败"" $email >>/tmp/mailx.log 2>&1
fi
```
3.4.5 配置编写VIP脚本
```
vim /etc/mha/scripts/master_ip_online_change
#!/bin/bash
source /root/.bash_profile
vip=`echo '10.180.160.118/24'` #设置VIP
key=`echo '1'`
command=`echo "$1" | awk -F = '{print $2}'`
orig_master_host=`echo "$2" | awk -F = '{print $2}'`
new_master_host=`echo "$7" | awk -F = '{print $2}'`
orig_master_ssh_user=`echo "${12}" | awk -F = '{print $2}'`
new_master_ssh_user=`echo "${13}" | awk -F = '{print $2}'`
#要求服务的网卡识别名一样，都为ens160(这里是)
stop_vip=`echo "ssh root@$orig_master_host /usr/sbin/ifconfig ens160:$key down"`
start_vip=`echo "ssh root@$new_master_host /usr/sbin/ifconfig ens160:$key $vip"`
if [ $command = 'stop' ]
then
echo -e "\n\n\n****************************\n"
echo -e "Disabled thi VIP - $vip on old master: $orig_master_host \n"
$stop_vip
if [ $? -eq 0 ]
then
echo "Disabled the VIP successfully"
else
echo "Disabled the VIP failed"
fi
echo -e "***************************\n\n\n"
fi
if [ $command = 'start' -o $command = 'status' ]
then
echo -e "\n\n\n*************************\n"
echo -e "Enabling the VIP - $vip on new master: $new_master_host \n"
$start_vip
if [ $? -eq 0 ]
then
echo "Enabled the VIP successfully"
else
echo "Enabled the VIP failed"
fi
echo -e "***************************\n\n\n"
fi
```
3.4.6将脚本赋予可执行权限
```
chmod +x /etc/mha/scripts/master_ip_failover
chmod +x /etc/mha/scripts/master_ip_online_change
chmod +x /etc/mha/scripts/send_report
```
3.4.7 通过 masterha_check_ssh 验证 ssh 信任登录是否成功
```
masterha_check_ssh --conf=/etc/mha/app1.cnf
Mon Nov 19 16:45:02 2018 - [debug] ok.
Mon Nov 19 16:45:02 2018 - [info] All SSH connection tests passed successfully. #表示所有都成功
```
3.4.8 通过 masterha_check_repl 验证 mysql 主从复制是否成功（下面输出表示测试通过）
```
masterha_check_repl --conf=/etc/mha/app1.cnf
IN SCRIPT TEST====/sbin/ifconfig ens160:1 down==/sbin/ifconfig ens160:1 10.180.160.118/24===
Checking the Status of the script.. OK
Tue Nov 20 09:44:11 2018 - [info] OK.
Tue Nov 20 09:44:11 2018 - [warning] shutdown_script is not defined.
Tue Nov 20 09:44:11 2018 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
```
3.5 启动MHA（此MHA监控脚本切换一次就会退出，需要手动再次启动）
3.5.1先在master上绑定vip，(只需要在master绑定这一次，以后会自动切换)
```
/usr/sbin/ifconfig ens160:1 10.180.160.118/24
```
3.5.2：然后通过 masterha_manager 启动 MHA 监控(在manager角色上执行)
```
mkdir /var/log/mha/app1 -p
touch /var/log/mha/app1/manager.log
```
#启动mha监控进程，下面是一条命令
```
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
```
#这样 MHA 的日志保存在/var/log/masterha/app1/manager.log 下
检查MHA的启动状态
tailf /var/log/mha/app1/manager.log
#如果最后一行是如下，表明启动成功
[info] Ping(SELECT) succeeded, waiting until MySQL doesn’t respond..
3.5.3 检查集群状态
masterha_check_status --conf=/etc/mha/app1.cnf
```
masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:15778) is running(0:PING_OK), master:10.180.160.113
```
**四,切换测试**
4.1 切换测试之：自动切换（模拟master宕了）(实验测试当物理故障时，没有指定shutdown_script是没用的，不切换)
4.1.1. 要实现自动 Failover,必须先启动 MHA Manager,否则无法自动切换
A、杀掉主库 mysql 进程,模拟主库发生故障,进行自动 failover 操作。
B、看 MHA 切换日志,了解整个切换过程
tail -f /var/log/mha/app1/manager.log
```
tail -fn1000 /var/log/mha/app1/manager.log
Tue Nov 20 13:57:20 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
Tue Nov 20 13:57:20 2018 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Tue Nov 20 13:57:20 2018 - [info] Reading server configuration from /etc/mha/app1.cnf..
Tue Nov 20 14:00:03 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
Tue Nov 20 14:00:03 2018 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Tue Nov 20 14:00:03 2018 - [info] Reading server configuration from /etc/mha/app1.cnf..
:3306)
Tue Nov 20 13:57:21 2018 - [info] 10.180.160.114(10.180.160.114:3306)
Tue Nov 20 13:57:21 2018 - [info] 10.180.160.115(10.180.160.115:3306)
Tue Nov 20 13:57:21 2018 - [info] Alive Slaves:
Tue Nov 20 13:57:21 2018 - [info] 10.180.160.114(10.180.160.114:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 13:57:21 2018 - [info] GTID ON
Tue Nov 20 13:57:21 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 13:57:21 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Nov 20 13:57:21 2018 - [info] 10.180.160.115(10.180.160.115:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 13:57:21 2018 - [info] GTID ON
Tue Nov 20 13:57:21 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 13:57:21 2018 - [info] Not candidate for the new Master (no_master is set)
Tue Nov 20 13:57:21 2018 - [info] Current Alive Master: 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 13:57:21 2018 - [info] Checking slave configurations..
Tue Nov 20 13:57:21 2018 - [info] Checking replication filtering settings..
Tue Nov 20 13:57:21 2018 - [info] binlog_do_db= , binlog_ignore_db=
Tue Nov 20 13:57:21 2018 - [info] Replication filtering check ok.
Tue Nov 20 13:57:21 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Tue Nov 20 13:57:21 2018 - [info] Checking SSH publickey authentication settings on the current master..
Tue Nov 20 13:57:21 2018 - [info] HealthCheck: SSH to 10.180.160.113 is reachable.
Tue Nov 20 13:57:21 2018 - [info]
10.180.160.113(10.180.160.113:3306) (current master)
+--10.180.160.114(10.180.160.114:3306)
+--10.180.160.115(10.180.160.115:3306)
Tue Nov 20 13:57:21 2018 - [info] Checking master_ip_failover_script status:
Tue Nov 20 13:57:21 2018 - [info] /etc/mha/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.180.160.113 --orig_master_ip=10.180.160.113 --orig_master_port=3306
IN SCRIPT TEST====/sbin/ifconfig ens160:1 down==/sbin/ifconfig ens160:1 10.180.160.118/24===
Checking the Status of the script.. OK
Tue Nov 20 13:57:21 2018 - [info] OK.
Tue Nov 20 13:57:21 2018 - [warning] shutdown_script is not defined.
Tue Nov 20 13:57:21 2018 - [info] Set master ping interval 1 seconds.
Tue Nov 20 13:57:21 2018 - [info] Set secondary check script: masterha_secondary_check -s 10.180.160.113 -s 10.180.160.114 -s 10.180.160.115
Tue Nov 20 13:57:21 2018 - [info] Starting ping health check on 10.180.160.113(10.180.160.113:3306)..
Tue Nov 20 13:57:21 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Tue Nov 20 13:59:59 2018 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Tue Nov 20 13:59:59 2018 - [info] Executing secondary network check script: masterha_secondary_check -s 10.180.160.113 -s 10.180.160.114 -s 10.180.160.115 --user=root --master_host=10.180.160.113 --master_ip=10.180.160.113 --master_port=3306 --master_user=root --master_password=dbaadmin@123 --ping_type=SELECT
Tue Nov 20 13:59:59 2018 - [info] Executing SSH check script: exit 0
Tue Nov 20 13:59:59 2018 - [info] HealthCheck: SSH to 10.180.160.113 is reachable.
Monitoring server 10.180.160.113 is reachable, Master is not reachable from 10.180.160.113. OK.
Monitoring server 10.180.160.114 is reachable, Master is not reachable from 10.180.160.114. OK.
Monitoring server 10.180.160.115 is reachable, Master is not reachable from 10.180.160.115. OK.
Tue Nov 20 14:00:00 2018 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Tue Nov 20 14:00:00 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.180.160.113' (111))
Tue Nov 20 14:00:00 2018 - [warning] Connection failed 2 time(s)..
Tue Nov 20 14:00:01 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.180.160.113' (111))
Tue Nov 20 14:00:01 2018 - [warning] Connection failed 3 time(s)..
Tue Nov 20 14:00:02 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.180.160.113' (111))
Tue Nov 20 14:00:02 2018 - [warning] Connection failed 4 time(s)..
Tue Nov 20 14:00:02 2018 - [warning] Master is not reachable from health checker!
Tue Nov 20 14:00:02 2018 - [warning] Master 10.180.160.113(10.180.160.113:3306) is not reachable!
Tue Nov 20 14:00:02 2018 - [warning] SSH is reachable.
Tue Nov 20 14:00:02 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1.cnf again, and trying to connect to all servers to check server status..
Tue Nov 20 14:00:02 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
Tue Nov 20 14:00:02 2018 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Tue Nov 20 14:00:02 2018 - [info] Reading server configuration from /etc/mha/app1.cnf..
Tue Nov 20 14:00:03 2018 - [info] GTID failover mode = 1
Tue Nov 20 14:00:03 2018 - [info] Dead Servers:
Tue Nov 20 14:00:03 2018 - [info] 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:03 2018 - [info] Alive Servers:
Tue Nov 20 14:00:03 2018 - [info] 10.180.160.114(10.180.160.114:3306)
Tue Nov 20 14:00:03 2018 - [info] 10.180.160.115(10.180.160.115:3306)
Tue Nov 20 14:00:03 2018 - [info] Alive Slaves:
Tue Nov 20 14:00:03 2018 - [info] 10.180.160.114(10.180.160.114:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:03 2018 - [info] GTID ON
Tue Nov 20 14:00:03 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:03 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Nov 20 14:00:03 2018 - [info] 10.180.160.115(10.180.160.115:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:03 2018 - [info] GTID ON
Tue Nov 20 14:00:03 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:03 2018 - [info] Not candidate for the new Master (no_master is set)
Tue Nov 20 14:00:03 2018 - [info] Checking slave configurations..
Tue Nov 20 14:00:03 2018 - [info] Checking replication filtering settings..
Tue Nov 20 14:00:03 2018 - [info] Replication filtering check ok.
Tue Nov 20 14:00:03 2018 - [info] Master is down!
Tue Nov 20 14:00:03 2018 - [info] Terminating monitoring script.
Tue Nov 20 14:00:03 2018 - [info] Got exit code 20 (Master dead).
Tue Nov 20 14:00:03 2018 - [info] MHA::MasterFailover version 0.58.
Tue Nov 20 14:00:03 2018 - [info] Starting master failover.
Tue Nov 20 14:00:03 2018 - [info]
Tue Nov 20 14:00:03 2018 - [info] * Phase 1: Configuration Check Phase..
Tue Nov 20 14:00:03 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] GTID failover mode = 1
Tue Nov 20 14:00:04 2018 - [info] Dead Servers:
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Checking master reachability via MySQL(double check)...
Tue Nov 20 14:00:04 2018 - [info] ok.
Tue Nov 20 14:00:04 2018 - [info] Alive Servers:
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.114(10.180.160.114:3306)
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.115(10.180.160.115:3306)
Tue Nov 20 14:00:04 2018 - [info] Alive Slaves:
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.114(10.180.160.114:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.115(10.180.160.115:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Not candidate for the new Master (no_master is set)
Tue Nov 20 14:00:04 2018 - [info] Starting GTID based failover.
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Nov 20 14:00:04 2018 - [info] Executing master IP deactivation script:
Tue Nov 20 14:00:04 2018 - [info] /etc/mha/scripts/master_ip_failover --orig_master_host=10.180.160.113 --orig_master_ip=10.180.160.113 --orig_master_port=3306 --command=stopssh --ssh_user=root
IN SCRIPT TEST====/sbin/ifconfig ens160:1 down==/sbin/ifconfig ens160:1 10.180.160.118/24===
Disabling the VIP on old master: 10.180.160.113
Tue Nov 20 14:00:04 2018 - [info] done.
Tue Nov 20 14:00:04 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Nov 20 14:00:04 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] * Phase 3: Master Recovery Phase..
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000001:1526
Tue Nov 20 14:00:04 2018 - [info] Retrieved Gtid Set: 2ccd6e03-2d96-11e8-9cfc-0050569666a2:1-6
Tue Nov 20 14:00:04 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.114(10.180.160.114:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.115(10.180.160.115:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Not candidate for the new Master (no_master is set)
Tue Nov 20 14:00:04 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000001:1526
Tue Nov 20 14:00:04 2018 - [info] Retrieved Gtid Set: 2ccd6e03-2d96-11e8-9cfc-0050569666a2:1-6
Tue Nov 20 14:00:04 2018 - [info] Oldest slaves:
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.114(10.180.160.114:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.115(10.180.160.115:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Not candidate for the new Master (no_master is set)
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] * Phase 3.3: Determining New Master Phase..
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] Searching new master from slaves..
Tue Nov 20 14:00:04 2018 - [info] Candidate masters from the configuration file:
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.114(10.180.160.114:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Nov 20 14:00:04 2018 - [info] Non-candidate masters:
Tue Nov 20 14:00:04 2018 - [info] 10.180.160.115(10.180.160.115:3306) Version=5.7.17-log (oldest major version between slaves) log-bin:enabled
Tue Nov 20 14:00:04 2018 - [info] GTID ON
Tue Nov 20 14:00:04 2018 - [info] Replicating from 10.180.160.113(10.180.160.113:3306)
Tue Nov 20 14:00:04 2018 - [info] Not candidate for the new Master (no_master is set)
Tue Nov 20 14:00:04 2018 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Tue Nov 20 14:00:04 2018 - [info] New master is 10.180.160.114(10.180.160.114:3306)
Tue Nov 20 14:00:04 2018 - [info] Starting master failover..
Tue Nov 20 14:00:04 2018 - [info]
From:
10.180.160.113(10.180.160.113:3306) (current master)
+--10.180.160.114(10.180.160.114:3306)
+--10.180.160.115(10.180.160.115:3306)
To:
10.180.160.114(10.180.160.114:3306) (new master)
+--10.180.160.115(10.180.160.115:3306)
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] * Phase 3.3: New Master Recovery Phase..
Tue Nov 20 14:00:04 2018 - [info]
Tue Nov 20 14:00:04 2018 - [info] Waiting all logs to be applied..
Tue Nov 20 14:00:04 2018 - [info] done.
Tue Nov 20 14:00:04 2018 - [info] Getting new master's binlog name and position..
Tue Nov 20 14:00:04 2018 - [info] mysql-bin.000001:1499
Tue Nov 20 14:00:04 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.180.160.114', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Tue Nov 20 14:00:04 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000001, 1499, 2ccd6e03-2d96-11e8-9cfc-0050569666a2:1-6
Tue Nov 20 14:00:04 2018 - [info] Executing master IP activate script:
Tue Nov 20 14:00:04 2018 - [info] /etc/mha/scripts/master_ip_failover --command=start --ssh_user=root --orig_master_host=10.180.160.113 --orig_master_ip=10.180.160.113 --orig_master_port=3306 --new_master_host=10.180.160.114 --new_master_ip=10.180.160.114 --new_master_port=3306 --new_master_user='root' --new_master_password=xxx
Unknown option: new_master_user
Unknown option: new_master_password
IN SCRIPT TEST====/sbin/ifconfig ens160:1 down==/sbin/ifconfig ens160:1 10.180.160.118/24===
Enabling the VIP - 10.180.160.118/24 on the new master - 10.180.160.114
Tue Nov 20 14:00:05 2018 - [info] OK.
Tue Nov 20 14:00:05 2018 - [info] Setting read_only=0 on 10.180.160.114(10.180.160.114:3306)..
Tue Nov 20 14:00:05 2018 - [info] ok.
Tue Nov 20 14:00:05 2018 - [info] ** Finished master recovery successfully.
Tue Nov 20 14:00:05 2018 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov 20 14:00:05 2018 - [info]
Tue Nov 20 14:00:05 2018 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov 20 14:00:05 2018 - [info]
Tue Nov 20 14:00:05 2018 - [info]
Tue Nov 20 14:00:05 2018 - [info] * Phase 4.1: Starting Slaves in parallel..
Tue Nov 20 14:00:05 2018 - [info]
Tue Nov 20 14:00:05 2018 - [info] -- Slave recovery on host 10.180.160.115(10.180.160.115:3306) started, pid: 14536. Check tmp log /var/log/mha/app1/10.180.160.115_3306_20181120140003.log if it takes time..
Tue Nov 20 14:00:07 2018 - [info]
Tue Nov 20 14:00:07 2018 - [info] Log messages from 10.180.160.115 ...
Tue Nov 20 14:00:07 2018 - [info]
Tue Nov 20 14:00:05 2018 - [info] Resetting slave 10.180.160.115(10.180.160.115:3306) and starting replication from the new master 10.180.160.114(10.180.160.114:3306)..
Tue Nov 20 14:00:05 2018 - [info] Executed CHANGE MASTER.
Tue Nov 20 14:00:06 2018 - [info] Slave started.
Tue Nov 20 14:00:06 2018 - [info] gtid_wait(2ccd6e03-2d96-11e8-9cfc-0050569666a2:1-6) completed on 10.180.160.115(10.180.160.115:3306). Executed 0 events.
Tue Nov 20 14:00:07 2018 - [info] End of log messages from 10.180.160.115.
Tue Nov 20 14:00:07 2018 - [info] -- Slave on host 10.180.160.115(10.180.160.115:3306) started.
Tue Nov 20 14:00:07 2018 - [info] All new slave servers recovered successfully.
Tue Nov 20 14:00:07 2018 - [info]
Tue Nov 20 14:00:07 2018 - [info] * Phase 5: New master cleanup phase..
Tue Nov 20 14:00:07 2018 - [info]
Tue Nov 20 14:00:07 2018 - [info] Resetting slave info on the new master..
Tue Nov 20 14:00:07 2018 - [info] 10.180.160.114: Resetting slave info succeeded.
Tue Nov 20 14:00:07 2018 - [info] Master failover to 10.180.160.114(10.180.160.114:3306) completed successfully.
Tue Nov 20 14:00:07 2018 - [info] Deleted server1 entry from /etc/mha/app1.cnf .
Tue Nov 20 14:00:07 2018 - [info]
----- Failover Report -----
app1: MySQL Master failover 10.180.160.113(10.180.160.113:3306) to 10.180.160.114(10.180.160.114:3306) succeeded
Master 10.180.160.113(10.180.160.113:3306) is down!
Check MHA Manager logs at kubernetes-113:/var/log/mha/app1/manager.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 10.180.160.113(10.180.160.113:3306)
Selected 10.180.160.114(10.180.160.114:3306) as a new master.
10.180.160.114(10.180.160.114:3306): OK: Applying all logs succeeded.
10.180.160.114(10.180.160.114:3306): OK: Activated master IP address.
10.180.160.115(10.180.160.115:3306): OK: Slave started, replicating from 10.180.160.114(10.180.160.114:3306)
10.180.160.114(10.180.160.114:3306): Resetting slave info succeeded.
Master failover to 10.180.160.114(10.180.160.114:3306) completed successfully.
```
4.1.2.从上面的输出可以看出整个 MHA 的切换过程,共包括以下的步骤：
1).配置文件检查阶段,这个阶段会检查整个集群配置文件配置
2).宕机的 master 处理,这个阶段包括虚拟 ip 摘除操作,主机关机操作（由于没有定义power_manager脚本，不会关机）
3).复制 dead maste 和最新 slave 相差的 relay log,并保存到 MHA Manger 具体的目录下
4).识别含有最新更新的 slave
5).应用从 master 保存的二进制日志事件(binlog events)（这点信息对于将故障master修复后加入集群很重要）
6).提升一个 slave 为新的 master 进行复制
7).使其他的 slave 连接新的 master 进行复制
4.1.3 切换完成后,关注如下变化:
1、vip 自动从原来的 master 切换到新的 master,同时,manager 节点的监控进程自动退出。
2、在日志目录(/var/log/mha/app1)产生一个 app1.failover.complete 文件
3、/etc/mha/app1.cnf 配置文件中原来老的 master 配置被删除。
4.2. 切换测试之：在线切换（用于硬件升级）
#MHA 在线切换是 MHA 除了自动监控切换换提供的另外一种方式,多用于诸如硬件升级，MySQL 数据库迁移等等。该方式提供快速切换和优雅的阻塞写入,无关关闭原有服务器，整个切换过程在 0.5-2s 的时间左右,大大减少了停机时间
4.2.1 注意点：前提，mha监控没有运行的情况下，才能进行
A、老master上的vip已经正确生效了
B、各个salve节点数据库的sql_IO和sql_sql进程都正常（即YES)
show slave status\G;
C、MHA脚本不能运行，若已处于监控状态，需要停掉它
masterha_stop --conf=/etc/mha/app1.cnf
4.2.2执行切换
#需要填写新的master的IP
masterha_master_switch --conf=/etc/mha/app1.cnf --master_state=alive --new_master_host=10.180.160.114 --orig_master_is_new_slave --running_updates_limit=10000 --interactive=0
4.2.3 MHA 在线切换基本步骤：
a、检测 MHA 配置置及确认当前 master
b、决定新的 master
c、阻塞写入到当前 master
d、等待所有从服务器与现有 master 完成同步
e、在新 master 授予写权限,以及并行切换从库
f、重置原 master 为新 master 的 slave
g、在线切换不会删除/etc/mha/app1.cnf 配置文件中原来老的 master 配置
4.3. 如何将故障节点重新加入集群（自动切换的情况下）
通常情况下自动切换以后,原 master 可能已经废弃掉,待原 master 主机修复后,如果数据完整的情况下,可能想把原来 master 重新作为新主库的 slave,这时我们可以借助当时自动切换时刻的 MHA 日志来完成对原 master 的修复。(若是下面第二步：出现的是第二种，可以不用借助日志，定位binlog日志点，可以自动定位)
4.3.1、修改 manager 配置文件（只针对自动切换的，在线切换不会删除配置）
#将如下内容添加到/etc/mha/app1.conf 中
[server1]
hostname=10.180.160.113
candidate_master=1
master_binlog_dir="/apps/mysql/binlogs"
4.3.2、修复老的 master,然后设置为 slave
从自动切换时刻的 MHA 日志上可以发现类似如下信息：
#意思是说,如果 Master 主机修复好了,可以在修复好后的 Master 上执行 CHANGE MASTER操作,作为新的 slave 库。
cat /var/log/mha/app1/manager.log
```
Tue Nov 20 14:00:04 2018 - [info] mysql-bin.000001:1499
Tue Nov 20 14:00:04 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.180.160.114', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Tue Nov 20 14:00:04 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000001, 1499, 2ccd6e03-2d96-11e8-9cfc-0050569666a2:1-6
Tue Nov 20 14:00:04 2018 - [info] Executing master IP activate script:
```
在老的 master 执行如下命令:(具体执行哪条，根据上面日志输出来确定，区别是一个有日志的定位，一个是自动定位)
```
mysql> change master to master_host='10.180.160.114',master_user='repl',master_password='repl@123',master_log_file='mysql-bin.000001',master_log_pos=1499;
#或则
mysql> change master to master_host='10.180.160.114',master_user='repl',master_password='repl@123',MASTER_AUTO_POSITION=1;
mysql> start slave;
mysql> show slave status\G;
```
#这样,数据就开始同步到老的 master 上了。此时老的 master 已经重新加入集群,变成 mha集群中的一个 slave 角色了。
设置老的master为只读
```
mysql -uroot -pdbaadmin@123 -e "set global read_only=1"
```
4.3.3、在 manger 节点上重新启动监控进程
```
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
```
扫一扫
566
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
