【postgresql】repmgr 自动 switchover 切换

1.集群状态检查 

[pgsql@pg2:/home/pgsql]$repmgr -f /postgresql/app/postgresql/repmgr.conf cluster show
 ID | Name | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string                                                      
----+------+---------+-----------+----------+----------+----------+----------+-------------------------------------------------------------------------
 1  | pg1  | standby |   running | pg2      | default  | 100      | 5        | host=192.168.1.10 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 2  | pg2  | primary | * running |          | default  | 100      | 5        | host=192.168.1.11 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 3  | pg3  | standby |   running | pg2      | default  | 100      | 5        | host=192.168.1.12 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 4  | pg4  | witness | * running | pg2      | default  | 0        | n/a      | host=192.168.1.13 port=5432 user=repmgr dbname=repmgr connect_timeout=2

2.关闭主库，模拟主库异常 

[pgsql@pg2:/home/pgsql]$pg_ctl stop 
waiting for server to shut down....... done
server stopped

[pgsql@pg4:/home/pgsql]$repmgr -f /postgresql/app/postgresql/repmgr.conf cluster show
 ID | Name | Role    | Status        | Upstream | Location | Priority | Timeline | Connection string                                                      
----+------+---------+---------------+----------+----------+----------+----------+-------------------------------------------------------------------------
 1  | pg1  | standby |   running     | ? pg2    | default  | 100      | 5        | host=192.168.1.10 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 2  | pg2  | primary | ? unreachable | ?        | default  | 100      |          | host=192.168.1.11 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 3  | pg3  | standby |   running     | ? pg2    | default  | 100      | 5        | host=192.168.1.12 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 4  | pg4  | witness | * running     | ? pg2    | default  | 0        | n/a      | host=192.168.1.13 port=5432 user=repmgr dbname=repmgr connect_timeout=2

WARNING: following issues were detected
  - unable to connect to node "pg1" (ID: 1)'s upstream node "pg2" (ID: 2)
  - unable to determine if node "pg1" (ID: 1) is attached to its upstream node "pg2" (ID: 2)
  - unable to connect to node "pg2" (ID: 2)
  - node "pg2" (ID: 2) is registered as an active primary but is unreachable
  - unable to connect to node "pg3" (ID: 3)'s upstream node "pg2" (ID: 2)
  - unable to determine if node "pg3" (ID: 3) is attached to its upstream node "pg2" (ID: 2)
  - unable to connect to node "pg4" (ID: 4)''s upstream node "pg2" (ID: 2)

HINT: execute with --verbose option to see connection error messages
[pgsql@pg4:/home/pgsql]$repmgr -f /postgresql/app/postgresql/repmgr.conf cluster show
 ID | Name | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string                                                      
----+------+---------+-----------+----------+----------+----------+----------+-------------------------------------------------------------------------
 1  | pg1  | primary | * running |          | default  | 100      | 6        | host=192.168.1.10 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 2  | pg2  | primary | - failed  | ?        | default  | 100      |          | host=192.168.1.11 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 3  | pg3  | standby |   running | pg1      | default  | 100      | 5        | host=192.168.1.12 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 4  | pg4  | witness | * running | pg1      | default  | 0        | n/a      | host=192.168.1.13 port=5432 user=repmgr dbname=repmgr connect_timeout=2

WARNING: following issues were detected
  - unable to connect to node "pg2" (ID: 2)

HINT: execute with --verbose option to see connection error messages

已经发送主备切换,原来的主变为failed; 

3.旧库加入集群 

pg_ctl start  

[pgsql@pg4:/home/pgsql]$repmgr -f /postgresql/app/postgresql/repmgr.conf cluster show
 ID | Name | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string                                                      
----+------+---------+-----------+----------+----------+----------+----------+-------------------------------------------------------------------------
 1  | pg1  | primary | * running |          | default  | 100      | 6        | host=192.168.1.10 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 2  | pg2  | primary | ! running |          | default  | 100      | 5        | host=192.168.1.11 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 3  | pg3  | standby |   running | pg1      | default  | 100      | 5        | host=192.168.1.12 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 4  | pg4  | witness | * running | pg1      | default  | 0        | n/a      | host=192.168.1.13 port=5432 user=repmgr dbname=repmgr connect_timeout=2

WARNING: following issues were detected
  - node "pg2" (ID: 2) is running but the repmgr node record is inactive

已经出现脑裂。
停止并加入集群 
pg_ctl stop ; 
--加入集群前检查 
repmgr -f /postgresql/app/postgresql/repmgr.conf node rejoin -d 'host=192.168.1.10 user=repmgr dbname=repmgr connect_timeout=2' --force-rewind --dry-run --verbose
--正式加入集群 
repmgr -f /postgresql/app/postgresql/repmgr.conf node rejoin -d 'host=192.168.1.10 user=repmgr dbname=repmgr connect_timeout=2' --force-rewind --verbose
DETAIL: rejoin target server''s timeline 6 forked off current database system timeline 5 before current recovery point 0/5E000028
INFO: prerequisites for using pg_rewind are met

时间线不对，无法加入。

4.重做备库

repmgr -h 192.168.1.10 -U repmgr -d repmgr -f /postgresql/app/postgresql/repmgr.conf standby clone -F
pg_ctl start 
repmgr -h 192.168.1.10 -U repmgr -d repmgr -f /postgresql/app/postgresql/repmgr.conf standby register -F

--重新克隆后检查 
[pgsql@pg4:/home/pgsql]$repmgr -f /postgresql/app/postgresql/repmgr.conf cluster show
 ID | Name | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string                                                      
----+------+---------+-----------+----------+----------+----------+----------+-------------------------------------------------------------------------
 1  | pg1  | primary | * running |          | default  | 100      | 6        | host=192.168.1.10 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 2  | pg2  | standby |   running | pg1      | default  | 100      | 6        | host=192.168.1.11 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 3  | pg3  | standby |   running | pg1      | default  | 100      | 6        | host=192.168.1.12 port=5432 user=repmgr dbname=repmgr connect_timeout=2
 4  | pg4  | witness | * running | pg1      | default  | 0        | n/a      | host=192.168.1.13 port=5432 user=repmgr dbname=repmgr connect_timeout=2
[pgsql@