orchestrator master故障检测切换输出

Aug 29 03:15:52 dba1 orchestrator[8092]: dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator: [mysql] 2021/08/29 03:15:52 packets.go:123: closing bad idle connection: EOF
Aug 29 03:15:52 dba1 orchestrator: 2021-08-29 03:15:52 ERROR dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator: [mysql] 2021/08/29 03:15:52 packets.go:123: closing bad idle connection: EOF
Aug 29 03:15:52 dba1 orchestrator: [mysql] 2021/08/29 03:15:52 packets.go:123: closing bad idle connection: EOF
Aug 29 03:15:52 dba1 orchestrator: 2021-08-29 03:15:52 ERROR dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator: 2021-08-29 03:15:52 ERROR dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator: 2021-08-29 03:15:52 ERROR dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator: 2021-08-29 03:15:52 ERROR dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator[8092]: dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator[8092]: dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator[8092]: dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator[8092]: dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:52 dba1 orchestrator: 2021-08-29 03:15:52 DEBUG writeInstance: will not update database_instance due to error: dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:55 dba1 orchestrator: 2021-08-29 03:15:55 DEBUG raft leader is 192.168.36.128:10008 (this host); state: Leader
Aug 29 03:15:57 dba1 orchestrator[8092]: DiscoverInstance(mysqldba1:3306) instance is nil in 0.003s (Backend: 0.002s, Instance: 0.001s), error=dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:57 dba1 orchestrator: 2021-08-29 03:15:57 WARNING DiscoverInstance(mysqldba1:3306) instance is nil in 0.003s (Backend: 0.002s, Instance: 0.001s), error=dial tcp 192.168.36.131:3306: connect: connection refused
Aug 29 03:15:58 dba1 orchestrator[8092]: executeCheckAndRecoverFunction: proceeding with UnreachableMaster detection on mysqldba1:3306; isActionable?: false; skipProcesses: false
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 DEBUG analysis: ClusterName: mysqldba1:3306, IsMaster: true, LastCheckValid: false, LastCheckPartialSuccess: false, CountReplicas: 2, CountValidReplicas: 2, CountValidReplicatingReplicas: 2, CountLaggingReplicas: 0, CountDelayedReplicas: 0, CountReplicasFailingToConnectToMaster: 0
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO executeCheckAndRecoverFunction: proceeding with UnreachableMaster detection on mysqldba1:3306; isActionable?: false; skipProcesses: false
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO topology_recovery: detected UnreachableMaster failure on mysqldba1:3306
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO topology_recovery: Running 1 OnFailureDetectionProcesses hooks
Aug 29 03:15:58 dba1 orchestrator[8092]: topology_recovery: detected UnreachableMaster failure on mysqldba1:3306
Aug 29 03:15:58 dba1 orchestrator[8092]: topology_recovery: Running 1 OnFailureDetectionProcesses hooks
Aug 29 03:15:58 dba1 orchestrator[8092]: auditType:emergently-read-topology-instance instance:mysqldba1:3306 cluster:mysqldba1:3306 message:UnreachableMaster
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO auditType:emergently-read-topology-instance instance:mysqldba1:3306 cluster:mysqldba1:3306 message:UnreachableMaster
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 DEBUG orchestrator/raft: applying command 3195: write-recovery-step
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO topology_recovery: Running OnFailureDetectionProcesses hook 1 of 1: echo ‘Detected UnreachableMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log
Aug 29 03:15:58 dba1 orchestrator[8092]: topology_recovery: Running OnFailureDetectionProcesses hook 1 of 1: echo ‘Detected UnreachableMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO auditType:emergently-read-topology-instance instance:mysqldba2:3306 cluster:mysqldba1:3306 message:UnreachableMaster
Aug 29 03:15:58 dba1 orchestrator[8092]: auditType:emergently-read-topology-instance instance:mysqldba2:3306 cluster:mysqldba1:3306 message:UnreachableMaster
Aug 29 03:15:58 dba1 orchestrator[8092]: auditType:emergently-read-topology-instance instance:mysqldba3:3306 cluster:mysqldba1:3306 message:UnreachableMaster
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO auditType:emergently-read-topology-instance instance:mysqldba3:3306 cluster:mysqldba1:3306 message:UnreachableMaster
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 DEBUG orchestrator/raft: applying command 3196: write-recovery-step
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO CommandRun(echo ‘Detected UnreachableMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log,[])
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO CommandRun/running: bash /tmp/orchestrator-process-cmd-535091933
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO CommandRun:
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO CommandRun successful. exit status 0
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO topology_recovery: Completed OnFailureDetectionProcesses hook 1 of 1 in 11.020349ms
Aug 29 03:15:58 dba1 orchestrator[8092]: CommandRun(echo ‘Detected UnreachableMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log,[])
Aug 29 03:15:58 dba1 orchestrator[8092]: CommandRun/running: bash /tmp/orchestrator-process-cmd-535091933
Aug 29 03:15:58 dba1 orchestrator[8092]: CommandRun:
Aug 29 03:15:58 dba1 orchestrator[8092]: CommandRun successful. exit status 0
Aug 29 03:15:58 dba1 orchestrator[8092]: topology_recovery: Completed OnFailureDetectionProcesses hook 1 of 1 in 11.020349ms
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 DEBUG orchestrator/raft: applying command 3197: write-recovery-step
Aug 29 03:15:58 dba1 orchestrator[8092]: topology_recovery: done running OnFailureDetectionProcesses hooks
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO topology_recovery: done running OnFailureDetectionProcesses hooks
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 DEBUG orchestrator/raft: applying command 3198: write-recovery-step
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 DEBUG orchestrator/raft: applying command 3199: register-failure-detection
Aug 29 03:15:58 dba1 orchestrator[8092]: executeCheckAndRecoverFunction: proceeding with UnreachableMaster recovery on mysqldba1:3306; isRecoverable?: false; skipProcesses: false
Aug 29 03:15:58 dba1 orchestrator: 2021-08-29 03:15:58 INFO executeCheckAndRecoverFunction: proceeding with UnreachableMaster recovery on mysqldba1:3306; isRecoverable?: false; skipProcesses: false
Aug 29 03:15:59 dba1 orchestrator[8092]: executeCheckAndRecoverFunction: ignoring analysisEntry that has no action plan: FirstTierReplicaFailingToConnectToMaster; key: mysqldba3:3306
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 DEBUG analysis: ClusterName: mysqldba1:3306, IsMaster: true, LastCheckValid: false, LastCheckPartialSuccess: false, CountReplicas: 2, CountValidReplicas: 2, CountValidReplicatingReplicas: 0, CountLaggingReplicas: 0, CountDelayedReplicas: 0, CountReplicasFailingToConnectToMaster: 2
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 WARNING executeCheckAndRecoverFunction: ignoring analysisEntry that has no action plan: FirstTierReplicaFailingToConnectToMaster; key: mysqldba3:3306
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO executeCheckAndRecoverFunction: proceeding with DeadMaster detection on mysqldba1:3306; isActionable?: true; skipProcesses: false
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 WARNING executeCheckAndRecoverFunction: ignoring analysisEntry that has no action plan: FirstTierReplicaFailingToConnectToMaster; key: mysqldba2:3306
Aug 29 03:15:59 dba1 orchestrator[8092]: executeCheckAndRecoverFunction: proceeding with DeadMaster detection on mysqldba1:3306; isActionable?: true; skipProcesses: false
Aug 29 03:15:59 dba1 orchestrator[8092]: executeCheckAndRecoverFunction: ignoring analysisEntry that has no action plan: FirstTierReplicaFailingToConnectToMaster; key: mysqldba2:3306
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO topology_recovery: detected DeadMaster failure on mysqldba1:3306
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO topology_recovery: Running 1 OnFailureDetectionProcesses hooks
Aug 29 03:15:59 dba1 orchestrator[8092]: topology_recovery: detected DeadMaster failure on mysqldba1:3306
Aug 29 03:15:59 dba1 orchestrator[8092]: topology_recovery: Running 1 OnFailureDetectionProcesses hooks
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 DEBUG orchestrator/raft: applying command 3200: write-recovery-step
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO topology_recovery: Running OnFailureDetectionProcesses hook 1 of 1: echo ‘Detected DeadMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log
Aug 29 03:15:59 dba1 orchestrator[8092]: topology_recovery: Running OnFailureDetectionProcesses hook 1 of 1: echo ‘Detected DeadMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 DEBUG orchestrator/raft: applying command 3201: write-recovery-step
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO CommandRun(echo ‘Detected DeadMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log,[])
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO CommandRun/running: bash /tmp/orchestrator-process-cmd-775705752
Aug 29 03:15:59 dba1 orchestrator[8092]: CommandRun(echo ‘Detected DeadMaster on mysqldba1:3306. Affected replicas: 2’ >> /tmp/recovery.log,[])
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO CommandRun:
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO CommandRun successful. exit status 0
Aug 29 03:15:59 dba1 orchestrator: 2021-08-29 03:15:59 INFO topology_recovery: Completed OnFailureDetectionProcesses hook 1 of 1 in 2.699262ms
Aug 29 03:15:59 dba1 orchestrator[8092]: CommandRun/running: bash /tmp/orchestrator-process-cmd-775705752
Aug 29 03:15:59 dba1 orchestrator[8092]: CommandRun:
Aug 29 03:15:59 dba1 orchestrator[8092]: CommandRun successful. exit status 0
Aug 29 03:15:59 dba1 orchestrator[8092]: topology_recovery: Completed OnFailureDetectionProces