orchestraror使用下篇

最新推荐文章于 2024-05-31 09:41:02 发布

叶落千尘

最新推荐文章于 2024-05-31 09:41:02 发布

阅读量1.5k

点赞数

分类专栏： mysql orchestrator 高可用文章标签： mysql 服务器数据库 orchestrator 高可用

本文链接：https://blog.csdn.net/baijiu1/article/details/112852093

版权

mysql 同时被 3 个专栏收录

33 篇文章 2 订阅

订阅专栏

高可用

6 篇文章 0 订阅

订阅专栏

orchestrator

5 篇文章 3 订阅

订阅专栏

概述

在之前，比较笼统的讲述了Orchestrator的大概部署与用法，本篇则详细描述Orchestrator的Hook、API命令的使用。还有其他部署过程中需要注意的点。
如果对Orchestrator还不太了解，可以参考：《 MySQL中间件:Orchestrator上篇》

配置细化

在说明Hook之前，要先了解Orchestrator是如何做故障检测的，以及是如何进行故障恢复的。

需要的配置

Orchestrator配置文件中的配置有一项：

{
   “ FailureDetectionPeriodBlockMinutes ”：60，
}
Orchestrator每秒探测一次。具体如何探测，下面会说。
FailureDetectionPeriodBlockMinutes：反垃圾邮件机制，阻止同一检测消息的重复发送

MySQL配置：
Orchestrator使用整体探测方式来检测是否故障，从而进行故障转移。
由于Orchestrator使用到MySQL本身的复制语句作为信息源，所以，需要在做主从复制的时候要配置网络心跳间隔和复制重试的相关参数，如下：

1、设置主从心跳超时时间：
set global slave_net_timeout = 4
如果在slave_net_timeout秒内没有收到来自master的任何数据，那么就会认为连接断开，进行重新连接。
2、设置重连次数与上限：
CHANGE MASTER TO MASTER_CONNECT_RETRY=1, MASTER_RETRY_COUNT=86400;
和上面的slave_net_timeout参数相关，如果slave_net_timeout秒内没有接收数据，那么会在MASTER_CONNECT_RETRY秒进行重新连接，直到MASTER_RETRY_COUNT上限制。

与传统探测机制的对比

对于一般探测的方式，一般的做法都是侦测主服务器，当无法查询或联系到主服务器时，发出警报，但是这种方式非常容易受到网络波动的影响。
由于上述方式容易受到网络波动，后来又增加了对服务器多次侦测和增加时间间隔，虽然减少了误报，但是增加了真正故障响应的时间。
Orchestrator解决了这些问题。它会在以下情况认为主机失效，要进行故障恢复：
1、连接master失败，无法连接到master，那么就判定master宕机。
2、能够连接到slave，并且在所有的slave上确认无法连接或检测到master主机，那么就判定master宕机。
从逻辑上来说，主连接不上之后，再去连接从库，在从库上去侦测主库，如果主库还是连接不上，那么断定主库宕机，然后进行故障恢复，是非常合理的。
对于一些特殊情况，Orchestrator并不会进行故障恢复：

在配置文件没有列出，或者没有配置故障恢复的集群
手动指定故障恢复不在特定的服务器上执行
手动禁用全局故障恢复设置
集群在不久前刚进行过故障恢复，在RecoveryPeriodBlockSeconds时间内
故障类型被认为是不值得故障恢复的(Orchestrator内置了非常多的故障策略，下面会细说)

Orchestrator中，探测和故障恢复两者没有关联，并且每秒都会进行一次探测，并且每次都会执行OnFailureDetectionProcesses钩子函数中的脚本。

故障恢复配置

Orchestrator故障恢复需要以下前提：
1、MySQL GTID(MASTER_AUTO_POSITION=1)
2、MariaDB GTID(logbin和log_slave_updates启用)
3、Pseudo GTID(logbin和log_slave_updates启用，并行复制设置slave_preserve_commit_order=1)
4、开启binlog日志(logbin和log_slave_updates启用)
有关故障恢复的配置如下：

{
   “ RecoveryPeriodBlockSeconds ”：3600，   --恢复后多少秒之内防止再次故障恢复
   “ RecoveryIgnoreHostnameFilters ”：[]，
   “ RecoverMasterClusterFilters ”：[      --要恢复集群的名称，也就是meta.cluster表中的集群名称
     “ thiscluster ”，
     “ thatcluster ”
  ]，
  “ RecoverIntermediateMasterClusterFilters ”：[    --恢复所有的集群
     “ * ”
  ]，
}
---
{
  "ApplyMySQLPromotionAfterMasterFailover": true,   --当为true时，Orchestrator将会在新的master上执行：reset slave all 和 set read_only=0两条命令
  "PreventCrossDataCenterMasterFailover": false,    --默认false，当为true时，Orchestrator只会使用同一个DC下的服务器进行恢复。如果在同一DC中找不到，那么恢复就会失败
  "PreventCrossRegionMasterFailover": false,       --默认false，当为true时，Orchestrator将会在同一区域内的服务器上进行故障恢复，如果找不到，那么恢复就会失败
  "FailMasterPromotionOnLagMinutes": 0,        
  "FailMasterPromotionIfSQLThreadNotUpToDate": true,   --如果所有从库都有延迟，那么提升的从库可能有未应用的中继日志，如果发出reset slave all语句，那么可能会丢失部分数据
  "DelayMasterPromotionIfSQLThreadNotUpToDate": false,   --当为true时，如果所有从库都有延迟，那么Orchestrator会等待SQL线程应用完毕，再提升新主
  "DetachLostReplicasAfterMasterFailover": true,     --某些从库再恢复过程中可能会丢失。当为true时，Orchestrator将通过detach-replica命令将其强行中断复制
  "MasterFailoverDetachReplicaMasterHost": false,   --当时true，Orchestrator将发出一个detach-replica-master-host命令到新的master上(确保新的master将不会复制旧的master)。 orchestrator将发出一个detach-replica-master-host升级的母版（这可以确保新的母版不会重生旧的母版）。默认值：false。如果ApplyMySQLPromotionAfterMasterFailover是true，那么该参数是无意义的。MasterFailoverDetachSlaveMasterHost是别名。
  "MasterFailoverLostInstancesDowntimeMinutes": 0,    --在故障转移之后，所有的服务器的停机分钟数。默认为0.
  "PostponeReplicaRecoveryOnLagMinutes": 0,    --在故障恢复过程中，复制的延迟超过给定时间的从库，只有在选出master并且执行了所有process流程之后，才会在恢复的后期进行恢复。0为禁用。PostponeSlaveRecoveryOnLagMinutes是别名。
}
---
 "OnFailureDetectionProcesses": [      --探测时执行的钩子
    "echo 'Detected {failureType} on {failureCluster}. Affected replicas: {countSlaves}' >> /tmp/recovery.log"
  ],
  "PreGracefulTakeoverProcesses": [    --在主变为只读之前执行
    "echo 'Planned takeover about to take place on {failureCluster}. Master will switch to read_only' >> /tmp/recovery.log"
  ],
  "PreFailoverProcesses": [            --在故障恢复之前立即执行，如果过程中有任何失败，那么整个恢复过程都会中止
    "echo 'Will recover from {failureType} on {failureCluster}' >> /tmp/recovery.log"
  ],
  "PostFailoverProcesses": [         --在故障恢复操作前执行
    "echo '(for all types) Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log",
    "/usr/local/src/orch_failover --dead_state={failureType} --vip_addr='10.211.55.100' --ssh_passwd='123456' --old_master={failedHost}:22 --interface='eth0' --new_master={successorHost}"
  ],
  "PostUnsuccessfulFailoverProcesses": [],     --在没有成功的故障恢复时执行
  "PostMasterFailoverProcesses": [         --在选举出新的master时执行
    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostIntermediateMasterFailoverProcesses": [    --在中间的主机上恢复，也就是级联复制的中间服务器
    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log"
  ],
  "PostGracefulTakeoverProcesses": [    --在老master放到新的master下时执行
    "echo 'Planned takeover complete' >> /tmp/recovery.log"
  ]

Orchestrator故障探测机制

Orchestrator有非常多的故障探测策略，对应到Hook里的{failureType}环境变量：
DeadMaster：
1、所有的master访问失败
2、所有的master的slave都复制失败
DeadMasterAndSomeReplicas：
1、所有的master访问失败
2、其中一些从库无法访问
3、其余的从库复制失败
DeadIntermediateMaster：
1、无法访问到级联复制中的中间服务器
2、中间服务器的所有从库都无法复制
UnreachableMasterWithLaggingReplicas：
1、无法访问到master
2、master的所有从库都有延迟
还有其他一些故障探测策略并不会触发故障恢复：
LockedSemiSyncMaster：
1、master正在启用半同步复制运行
2、连接半同步的从库少于rpl_semi_sync_master_wait_for_slave_count
3、rpl_semi_sync_master_timeout足够高，以至于主写入锁定不会退回异步复制
UnreachableMaster：
1、master访问失败
2、但是该master还有其他复制的从库

Orchestrator故障恢复

对于恢复过程，Orchestrator有以下几种故障恢复方案：

1、自动故障恢复(RecoverMasterClusterFilters 和RecoverIntermediateMasterClusterFilters参数控制恢复的集群或集群名称)
2、优雅的故障恢复(出于升级或维护目的，而暂时停止当前主机，用新主机去替换)，可使用以下命令：
orchestrator-client -c graceful-master-takeover -alias mycluster -d master:3306 指定从库，在降级的主机上不会开启复制
orchestrator-client -c graceful-master-takeover-auto -alias mycluster -d master:3306 指定从库，在降级的主机上开始复制
orchestrator-client -c graceful-master-takeover-auto -alias mycluster 让Orchestrator选择指定的副本升级，在降级的主服务器上开启复制
3、手动故障恢复(将实例指定为失败的复制)：
orchestrator-client -c recover -i dead_instance:3306 --debug
4、强制故障转移(无论什么，都进行转移)：
orchestrator-client -c force-master-failover --alias mycluster 或者
orchestrator-client -c force-master-failover -i instance.in.that.cluster

整个恢复过程由一系列的事件组成，包括：

1、恢复前Hook
2、拓补修复(由Orchestrator本身进行管理，用户无法配置)
3、恢复后Hook

在有些时候，某台服务器可能更适合于被提升，可以通过以下命令来提升指定的服务器恢复优先级：

orchestrator-client -c register-candidate -i instance --promotion-rule ${promotion_rule}

promotion_rule规则：

1、prefer      #偏向于这台
2、neutral     #中立(默认)
3、prefer_not  #不建议
4、must_not    #必须不能是这台

这些规则由一个小时的有效时间，在一个小时之后会失效，所以，要设置一个crontab的任务，来保持提升规则：

*/2 * * * * root "/usr/bin/perl -le 'sleep rand 10' && /usr/bin/orchestrator-client -c register-candidate -i hostname --promotion-rule prefer"

Downtime：
有时候出现了某些问题，我们要暂时下架某台机器或不想在故障恢复的时候恢复这台机器的时候，需要为这台执行一个命令，使其进入停机时间：
orchestrator-client -c begin-downtime -i instance:port

Orchestrator Hook

Orchestrator的Hook都是一系列的脚本组成的，这些脚本可以是shell脚本、python脚本、perl脚本等。总之，最后都是通过bash 命令来调用的，所以在编写的时候要注意这点。

Orchestrator API

orchestrator自带的客户端工具提供了非常多有用的API命令，下面介绍几个比较常用的命令：

orchestrator-client --help

[root@slave bin]# ./orchestrator-client --help
Usage: orchestrator-client -c [flags…]
Example: orchestrator-client -c which-master -i some.replica
Options:

-h, --help
print this help
-c , --command
indicate the operation to perform (see listing below)
-a , --alias
cluster alias
-o , --owner
name of owner for downtime/maintenance commands
-r , --reason
reason for downtime/maintenance operation
-u , --duration
duration for downtime/maintenance operations
-R , --promotion-rule
rule for ‘register-candidate’ command
-U <orchestrator_api>, --api <orchestrator_api>
override $orchestrator_api environemtn variable,
indicate where the client should connect to.
-P , --path
With ‘-c api’, indicate the specific API path you wish to call
-b username:password, --auth username:password
Specify when orchestrator uses basic HTTP auth.
-q , --query
Indicate query for ‘restart-replica-statements’ command
-l , --pool
pool name for pool related commands
-H -h
indicate host for resolve and raft operations

help                                     Show available commands
which-api                                Output the HTTP API to be used
api                                      Invoke any API request; provide --path argument
async-discover                           Lookup an instance, investigate it asynchronously. Useful for bulk loads
discover                                 Lookup an instance, investigate it
forget                                   Forget about an instance's existence
forget-cluster                           Forget about a cluster
topology                                 Show an ascii-graph of a replication topology, given a member of that topology
topology-tabulated                       Show an ascii-graph of a replication topology, given a member of that topology, in tabulated format
snapshot-topologies                      Trigger topology snapshot (recording host/master settings for all hosts)
clusters                                 List all clusters known to orchestrator
clusters-alias                           List all clusters known to orchestrator
search                                   Search for instances matching given substring
instance"|"which-instance                Output the fully-qualified hostname:port representation of the given instance, or error if unknown
which-master                             Output the fully-qualified hostname:port representation of a given instance's master
which-replicas                           Output the fully-qualified hostname:port list of replicas of a given instance
which-broken-replicas                    Output the fully-qualified hostname:port list of broken replicas of a given instance
which-cluster-instances                  Output the list of instances participating in same cluster as given instance
which-cluster                            Output the name of the cluster an instance belongs to, or error if unknown to orchestrator
which-cluster-alias                      Output the alias of the cluster an instance belongs to, or error if unknown to orchestrator
which-cluster-master                     Output the name of a writable master in given cluster
all-clusters-masters                     List of writeable masters, one per cluster
all-instances                            The complete list of known instances
which-cluster-osc-replicas               Output a list of replicas in a cluster, that could serve as a pt-online-schema-change operation control replicas
which-cluster-osc-running-replicas       Output a list of healthy, replicating replicas in a cluster, that could serve as a pt-online-schema-change operation control replicas
downtimed                                List all downtimed instances
dominant-dc                              Name the data center where most masters are found
submit-masters-to-kv-stores              Submit a cluster's master, or all clusters' masters to KV stores
relocate                                 Relocate a replica beneath another instance
relocate-replicas                        Relocates all or part of the replicas of a given instance under another instance
match                                    Matches a replica beneath another (destination) instance using Pseudo-GTID
match-up                                 Transport the replica one level up the hierarchy, making it child of its grandparent, using Pseudo-GTID
match-up-replicas                        Matches replicas of the given instance one level up the topology, making them siblings of given instance, using Pseudo-GTID
move-up                                  Move a replica one level up the topology
move-below                               Moves a replica beneath its sibling. Both replicas must be actively replicating from same master.
move-equivalent                          Moves a replica beneath another server, based on previously recorded "equivalence coordinates"
move-up-replicas                         Moves replicas of the given instance one level up the topology
make-co-master                           Create a master-master replication. Given instance is a replica which replicates directly from a master.
take-master                              Turn an instance into a master of its own master; essentially switch the two.
move-gtid                                Move a replica beneath another instance via GTID
move-replicas-gtid                       Moves all replicas of a given instance under another (destination) instance using GTID
repoint                                  Make the given instance replicate from another instance without changing the binglog coordinates. Use with care
repoint-replicas                         Repoint all replicas of given instance to replicate back from the instance. Use with care
take-siblings                            Turn all siblings of a replica into its sub-replicas.
tags                                     List tags for a given instance
tag-value                                List tags for a given instance
tag                                      Add a tag to a given instance. Tag in "tagname" or "tagname=tagvalue" format
untag                                    Remove a tag from an instance
untag-all                                Remove a tag from all matching instances
tagged                                   List instances tagged by tag-string. Format: "tagname" or "tagname=tagvalue" or comma separated "tag0,tag1=val1,tag2" for intersection of all.
submit-pool-instances                    Submit a pool name with a list of instances in that pool
which-heuristic-cluster-pool-instances   List instances of a given cluster which are in either any pool or in a specific pool
begin-downtime                           Mark an instance as downtimed
end-downtime                             Indicate an instance is no longer downtimed
begin-maintenance                        Request a maintenance lock on an instance
end-maintenance                          Remove maintenance lock from an instance
register-candidate                       Indicate the promotion rule for a given instance
register-hostname-unresolve              Assigns the given instance a virtual (aka "unresolved") name
deregister-hostname-unresolve            Explicitly deregister/dosassociate a hostname with an "unresolved" name
stop-replica                             Issue a STOP SLAVE on an instance
stop-replica-nice                        Issue a STOP SLAVE on an instance, make effort to stop such that SQL thread is in sync with IO thread (ie all relay logs consumed)
start-replica                            Issue a START SLAVE on an instance
restart-replica                          Issue STOP and START SLAVE on an instance
reset-replica                            Issues a RESET SLAVE command; use with care
detach-replica                           Stops replication and modifies Master_Host into an impossible yet reversible value.
reattach-replica                         Undo a detach-replica operation
detach-replica-master-host               Stops replication and modifies Master_Host into an impossible yet reversible value.
reattach-replica-master-host             Undo a detach-replica-master-host operation
skip-query                               Skip a single statement on a replica; either when running with GTID or without
which-gtid-errant                        Get errant GTID set (empty results if no errant GTID)
locate-gtid-errant                       List binary logs containing errant GTID
gtid-errant-reset-master                 Remove errant GTID transactions by way of RESET MASTER
gtid-errant-inject-empty                 Apply errant GTID as empty transactions on cluster's master
enable-semi-sync-master                  Enable semi-sync (master-side)
disable-semi-sync-master                 Disable semi-sync (master-side)
enable-semi-sync-replica                 Enable semi-sync (replica-side)
disable-semi-sync-replica                Disable semi-sync (replica-side)
restart-replica-statements               Given `-q "<query>"` that requires replication restart to apply, wrap query with stop/start slave statements as required to restore instance to same replication state. Print out set of statements
can-replicate-from                       Check if an instance can potentially replicate from another, according to replication rules
can-replicate-from-gtid                  Check if an instance can potentially replicate from another, according to replication rules and assuming Oracle GTID
is-replicating                           Check if an instance is replicating at this time (both SQL and IO threads running)
is-replication-stopped                   Check if both SQL and IO threads state are both strictly stopped.
set-read-only                            Turn an instance read-only, via SET GLOBAL read_only := 1
set-writeable                            Turn an instance writeable, via SET GLOBAL read_only := 0
flush-binary-logs                        Flush binary logs on an instance
purge-binary-logs                        Purge binary logs on an instance
last-pseudo-gtid                         Dump last injected Pseudo-GTID entry on a server
recover                                  Do auto-recovery given a dead instance, assuming orchestrator agrees there's a problem. Override blocking.
graceful-master-takeover                 Gracefully promote a new master. Either indicate identity of new master via '-d designated.instance.com' or setup replication tree to have a single direct replica to the master.
force-master-failover                    Forcibly discard master and initiate a failover, even if orchestrator doesn't see a problem. This command lets orchestrator choose the replacement master
force-master-takeover                    Forcibly discard master and promote another (direct child) instance instead, even if everything is running well
ack-cluster-recoveries                   Acknowledge recoveries for a given cluster; this unblocks pending future recoveries
ack-all-recoveries                       Acknowledge all recoveries
disable-global-recoveries                Disallow orchestrator from performing recoveries globally
enable-global-recoveries                 Allow orchestrator to perform recoveries globally
check-global-recoveries                  Show the global recovery configuration
replication-analysis                     Request an analysis of potential crash incidents in all known topologies
raft-leader                              Get identify of raft leader, assuming raft setup
raft-health                              Whether node is part of a healthy raft group
raft-leader-hostname                     Get hostname of raft leader, assuming raft setup
raft-elect-leader                        Request raft re-elections, provide hint for new leader's identity

[root@slave bin]#

显示所有集群及其别名：
[root@slave bin]# orchestrator-client -c clusters-alias
只显示集群名称：
[root@slave bin]# orchestrator-client -c clusters-alias | cut -d"," -f2 | sort
打印集群中的master实例：
[root@slave bin]# orchestrator-client -c which-cluster-master -alias mycluster
打印集群中的所有实例：
[root@slave bin]# orchestrator-client -c which-cluster-instances -alias mycluster
迁移从库slave1到另一个slave2实例下：
[root@slave bin]# orchestrator-client -c relocate -i slave1:3306 -d slave2: 3306
标记实例进入停机模式：
orchestrator-client -c begin-downtime -i slave1: 3306

最后

Orchestrator的探测机制在MySQL方面做了非常多的工作，在生产实践中也被证明是非常可靠的。
大多数的客户端命令，都可以在web界面上用鼠标实现。
如果修改了Orchestrator的端口，那么需要设置环境变量来指引客户端的API执行：

export ORCHESTRATOR_API="https://host1:13000/api https://host2:13000/api https://host3:13000/api"

目前对于传统复制，是不受支持的，所以在使用时一定要开启GTID模式，如果不想开启，那么可以使用Orchestrator自带的伪GTID，可以通过以下配置开启：

{
  "AutoPseudoGTID": true,
}

开启伪GTID的另一种方式是通过手动注入，可以通过以下配置开启：

{
  "PseudoGTIDPattern": "drop view if exists `meta`.`_pseudo_gtid_hint__asc:",
  "PseudoGTIDPatternIsFixedSubstring": true,
  "PseudoGTIDMonotonicHint": "asc:",
  "DetectPseudoGTIDQuery": "select count(*) as pseudo_gtid_exists from meta.pseudo_gtid_status where anchor = 1 and time_generated > now() - interval 2 hour",
}

如果有其他需求，可以看下我二次开发的Orchestrator：
https://gitee.com/haru_hi/orchestrator
目前新增：
1、双主切换
2、去除需要在切换集群中创建meta.cluster表中的标识机房字段，使用统一元数据库管理。
3、增加可允许最大延迟参数，为日志补齐做铺垫
4、修复优雅切换下不会选择同机房实例的问题
5、一键precheck和切换工具
6、vip漂移脚本
欢迎交流，私信我