基于pgpool-II读写分离+postgresql10主从从流复制高可用部署方案

一、环境准备

1、准备3台centos服务器：

pgpool-II版本：pgpool-II-pg10-4.2.0-1pgdg

postgresql版本：postgresql10

Server01：10.88.66.194 pgpoll-II(active/master) + PostgreSQL(primary/master)
Server02：10.88.66.195 pgpoll-II(standby/slave) + PostgreSQL(standby/slave)

Server03：10.88.66.196 pgpoll-II(standby/slave) + PostgreSQL(standby/slave)

二、3台节点安装postgresql、配置主从同步

1、安装postgresql yum源：

[all servers]# yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

[all servers]# yum install -y postgresql10-server

2、创建数据目录

[all servers]# mkdir -p /mnt/postgresql/data

[all servers]#chown -R postgres.postgres /mnt/postgresql/data

3、修改systemctl下的pgsql安装路径

[all servers]# vim /usr/lib/systemd/system/postgresql-10.service

Environment=PGDATA=/mnt/postgresql/data/

[all servers]# systemctl daemon-reload

4、初始化数据库：

[all servers]#su - postgres

[all servers]$/usr/pgsql-10/bin/initdb -D /mnt/postgresql/data

[all servers]$ mkdir /mnt/postgresql/data/archivedir ###创建wal归档目录

5、3台节点配置免密登录

[all servers]# passwd postgres

[all servers]# ssh-keygen -t rsa -f id_rsa_pgpool

[all servers]# ssh-copy-id -i id_rsa_pgpool.pub postgres@server1

[all servers]# ssh-copy-id -i id_rsa_pgpool.pub postgres@server2

[all servers]# ssh-copy-id -i id_rsa_pgpool.pub postgres@server3

[all servers]#su - postgres

[all servers]$ mkdir ~/.ssh

[all servers]$ chmod 700 ~/.ssh

[all servers]$ cd ~/.ssh

[all servers]$ ssh-keygen -t rsa -f id_rsa_pgpool

[all servers]$ ssh-copy-id -i id_rsa_pgpool.pub postgres@server1

[all servers]$ ssh-copy-id -i id_rsa_pgpool.pub postgres@server2

[all servers]$ ssh-copy-id -i id_rsa_pgpool.pub postgres@server3

6、修改主库pg_hba.conf配置文件,开启远程连接：

[Server01]#：vim /mnt/postgresql/data/pg_hba.conf

Server01#：vim /mnt/postgresql/data/postgresql.conf

简要说明：

#listen_address：*号表示所有主机都能访问，可按需修改

#wal_level：流复制模式要设为replica

#archive_mode：开启归档模式

#archive_command：wal日志归档的命令，归档的目录需要提前建好

#max_wal_senders：发送wal日志的进程数，需要大于从库的个数并且比max_connection参数值小

#wal_keep_segments：wal日志保留的个数，每个16M，用来保证主从的数据一致性

#hot_standby：控制在数据恢复期间，保持只读状态

7、启动主库数据库服务,修改密码postgres密码，创建流复制用户。

[Server01]#systemctl start postgresql-10.service

[Server01]#su - postgres

[postgres@Server01]$psql -U postgres -p 5432

postgres=# CREATE ROLE pgpool WITH LOGIN;

postgres=# CREATE ROLE repl WITH REPLICATION LOGIN;

postgres=# \password pgpool

postgres=# \password repl

postgres=# \password postgres

8、从库配置

[Server02]#rm -rf /mnt/postgresql/data/* ###清空数数据目录

[Server02]#pg_basebackup -h 10.88.66.194 -U postgres -F p -X stream -P -R -p5432 -D /mnt/postgresql/data/ -l backup202207012 ###拉取主库数据

[Server03]#rm -rf /mnt/postgresql/data/* ###清空数数据目录

[Server03]#pg_basebackup -h 10.88.66.194 -U postgres -F p -X stream -P -R -p5432 -D /mnt/postgresql/data/ -l backup202207012 ###拉取主库数据

参数简要说明：

-h：指定连接的数据库的主机名或ip地址，这里是主库的ip地址

-U：指定连接数据库的用户名，这里是之前创建的复制账号

-p：指定端口号

-F：指定生成的备份数据格式，p代表原样输出，t代表tar格式输出

-X：表示备份开始后，启动一个流复制连接从主库接收wal日志，有f（fetch）和s（stream）两种模式，建议使用s

-v：启用verbose模式，打印各阶段的日志

-P：显示数据文件、表空间传输的百分比，相当于备份进度

-R：备份完成之后自动生成recover.conf文件

-D：指定备份的目录，该目录在备份前需要手动清空

-l：指定备份的标识

9、修改recover.conf文件

10、修改data目录权限，启动数据库。 ###到这主从同步配置完成。

[Server02]#chown -R postgres.postgres /mnt/postgresql/data

[Server03]#chown -R postgres.postgres /mnt/postgresql/data

[Server02]#systemctl start postgresql-10.service

[Server03]#systemctl start postgresql-10.service

三、安装配置pgpool-II集群

pgpool官方下载地址：Index of /yum/rpms/4.2/redhat/rhel-7-x86_64

1、wget所用软件包

pgpool-II-pg10-4.2.0-1pgdg.rhel7.x86_64.rpm

pgpool-II-pg10-debuginfo-4.2.0-1pgdg.rhel7.x86_64.rpm

pgpool-II-pg10-devel-4.2.0-1pgdg.rhel7.x86_64.rpm

pgpool-II-pg10-extensions-4.2.0-1pgdg.rhel7.x86_64.rpm

2、安装pgpool

[all servers]#yum localinstall pgpool-II*

[all servers]#chown -R postgres.postgres /etc/pgpool-II/

[all servers]# su - postgres

[all servers]$ vi /var/lib/pgsql/.pgpass

server1:5432:postgres:postgres:<postgres user password>

server2:5432:postgres:postgres:<postgres user password>

server3:5432:postgres:postgres:<postgres user password>

[all servers]$ chmod 600 /var/lib/pgsql/.pgpass

3、创建一个pgpool_node_id文件并指定 pgpool（看门狗）节点号

[server01]# echo 0 > /etc/pgpool-II/pgpool_node_id

[server02]# echo 1 > /etc/pgpool-II/pgpool_node_id

[server03]# echo 2 > /etc/pgpool-II/pgpool_node_id

4、创建.pgpoolkey文件

[all servers]# su - postgres

[all servers]$ echo 'some string' > ~/.pgpoolkey

[all servers]$ chmod 600 ~/.pgpoolkey

5、在文件pool_passwd中注册用户名和AES加密密码

[all servers]# su - postgres

[all servers]$ pg_enc -m -k ~/.pgpoolkey -u pgpool -p

db password: [pgpool user's password]

[all servers]$ pg_enc -m -k ~/.pgpoolkey -u postgres -p

db password: [postgres user's password]

[all servers]$ cat /etc/pgpool-II/pool_passwd

pgpool:AESheq2ZMZjynddMWk5sKP/Rw==

postgres:AESHs/pWL5rtXy2IwuzroHfqg==

6、使用pg_md5为pgpool用户创建加密密码条目

[all servers]# echo 'pgpool:'`pg_md5 PCP password` >> /etc/pgpool-II/pcp.conf

7、修改pgpool主配置文件

[server01]# cat pgpool.conf | egrep -v "^\s*#" | grep -Ev "^$"

[server01]# vim /etc/pgpool-II/pgpool.conf

backend_clustering_mode = 'streaming_replication'

listen_addresses = '*'

port = 9999

socket_dir = '/var/run/postgresql'

reserved_connections = 0

pcp_listen_addresses = '*'

pcp_port = 9898

pcp_socket_dir = '/var/run/postgresql'

listen_backlog_multiplier = 2

serialize_accept = off

backend_hostname0 = 'server01'

backend_port0 = 5432

backend_weight0 = 1

backend_data_directory0 = '/mnt/postgresql/data'

backend_flag0 = 'ALLOW_TO_FAILOVER'

backend_application_name0 = 'server01'

backend_hostname1 = 'server02'

backend_port1 = 5432

backend_weight1 = 1

backend_data_directory1 = '/mnt/postgresql/data'

backend_flag1 = 'ALLOW_TO_FAILOVER'

backend_application_name1 = 'server02'

backend_hostname2 = 'server03'

backend_port2 = 5432

backend_weight2 = 1

backend_data_directory2 = '/mnt/postgresql/data'

backend_flag2 = 'ALLOW_TO_FAILOVER'

backend_application_name2 = 'server03'

enable_pool_hba = on

pool_passwd = 'pool_passwd'

authentication_timeout = 1min

allow_clear_text_frontend_auth = off

ssl = off

ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'

ssl_prefer_server_ciphers = off

ssl_ecdh_curve = 'prime256v1'

ssl_dh_params_file = ''

num_init_children = 32

max_pool = 4

child_life_time = 5min

child_max_connections = 0

connection_life_time = 0

client_idle_limit = 0

log_destination = 'stderr'

log_line_prefix = '%t: pid %p: ' # printf-style string to output at beginning of each log line.

log_connections = off

log_disconnections = off

log_hostname = off

log_statement = off

log_per_node_statement = off

log_client_messages = off

log_standby_delay = 'if_over_threshold'

syslog_facility = 'LOCAL0'

syslog_ident = 'pgpool'

logging_collector = on                          # Enable capturing of stderr

log_directory = '/var/log/pgpool_log'       # directory where log files are written,

log_filename = 'pgpool-%Y-%m-%d_%H%M%S.log'

log_truncate_on_rotation = on            # If on, an existing log file with the

log_rotation_age = 1d                        # Automatic rotation of logfiles will

log_rotation_size = 10MB                  # Automatic rotation of logfiles will

pid_file_name = '/var/run/pgpool/pgpool.pid'

logdir = '/tmp'

connection_cache = on

reset_query_list = 'ABORT; DISCARD ALL'

replicate_select = off

insert_lock = off

lobj_lock_table = ''

replication_stop_on_mismatch = off

failover_if_affected_tuples_mismatch = off

load_balance_mode = on

ignore_leading_white_space = on

read_only_function_list = ''

write_function_list = ''

primary_routing_query_pattern_list = ''

database_redirect_preference_list = ''

app_name_redirect_preference_list = ''

allow_sql_comments = off

disable_load_balance_on_write = 'transaction'

dml_adaptive_object_relationship_list= ''

statement_level_load_balance = off

sr_check_period = 10

sr_check_user = 'pgpool'

sr_check_password = ''

sr_check_database = 'postgres'

delay_threshold = 10000000

follow_primary_command = '/etc/pgpool-II/follow_primary.sh %d %h %p %D %m %H %M %P %r %R'

health_check_period = 5

health_check_timeout = 30

health_check_user = 'pgpool'

health_check_password = ''

health_check_database = ''

health_check_max_retries = 3

health_check_retry_delay = 1

connect_timeout = 10000

failover_command = '/etc/pgpool-II/failover.sh %d %h %p %D %m %H %M %P %r %R %N %S'

failback_command = ''

failover_on_backend_error = on

detach_false_primary = off

search_primary_node_timeout = 5min

recovery_user = 'nobody'

recovery_password = ''

recovery_1st_stage_command = ''

recovery_2nd_stage_command = ''

recovery_timeout = 90

client_idle_limit_in_recovery = 0

auto_failback = off

auto_failback_interval = 1min

use_watchdog = on

trusted_servers = ''

ping_path = '/bin'

hostname0 = 'server01'

wd_port0 = 9000

pgpool_port0 = 9999

hostname1 = 'server02'

wd_port1 = 9000

pgpool_port1 = 9999

hostname2 = 'server03'

wd_port2 = 9000

pgpool_port2 = 9999

wd_priority = 1

wd_authkey = ''

wd_ipc_socket_dir = '/var/run/postgresql'

delegate_IP = '10.88.66.228'

if_cmd_path = '/sbin'

if_up_cmd = '/usr/bin/sudo /sbin/ip addr add $_IP_$/24 dev enp59s0f0 label enp59s0f0:0'

if_down_cmd = '/usr/bin/sudo /sbin/ip addr del $_IP_$/24 dev enp59s0f0'

arping_path = '/usr/sbin'

arping_cmd = '/usr/bin/sudo /usr/sbin/arping -U $_IP_$ -w 1 -I enp59s0f0'

clear_memqcache_on_escalation = on

wd_escalation_command = '/etc/pgpool-II/escalation.sh'

wd_de_escalation_command = ''

failover_when_quorum_exists = on

failover_require_consensus = on

allow_multiple_failover_requests_from_node = off

enable_consensus_with_half_votes = off

wd_monitoring_interfaces_list = '' # Comma separated list of interfaces names to monitor.

wd_lifecheck_method = 'heartbeat'

wd_interval = 10

heartbeat_hostname0 = 'server01'

heartbeat_port0 = 9694

heartbeat_device0 = ''

heartbeat_hostname1 = 'server02'

heartbeat_port1 = 9694

heartbeat_device1 = ''

heartbeat_hostname2 = 'server03'

heartbeat_port2 = 9694

heartbeat_device2 = ''

wd_heartbeat_keepalive = 2

wd_heartbeat_deadtime = 30

wd_life_point = 3

wd_lifecheck_query = 'SELECT 1'

wd_lifecheck_dbname = 'template1'

wd_lifecheck_user = 'nobody'

wd_lifecheck_password = ''

relcache_expire = 0

relcache_size = 256

check_temp_table = catalog

check_unlogged_table = on

enable_shared_relcache = on

relcache_query_target = primary # Target node to send relcache queries. Default is primary node.

memory_cache_enabled = off

memqcache_method = 'shmem'

memqcache_memcached_host = 'localhost'

memqcache_memcached_port = 11211

memqcache_total_size = 64MB

memqcache_max_num_cache = 1000000

memqcache_expire = 0

memqcache_auto_cache_invalidation = on

memqcache_maxcache = 400kB

memqcache_cache_block_size = 1MB

memqcache_oiddir = '/var/log/pgpool/oiddir'

cache_safe_memqcache_table_list = ''

cache_unsafe_memqcache_table_list = ''

8、增加pool_hba.conf

host all pgpool 0.0.0.0/0 md5

host all postgres 0.0.0.0/0 md5

9、创建日志目录

[all servers]#mkdir /var/log/pgpool_log

[all servers]#chown -R postgres.postgres /var/log/pgpool_log/

10、同步配置文件

[server01]# scp /etc/pgpool-II/pgpool.conf root@server2:/etc/pgpool-II/pgpool.conf

[server02]# scp /etc/pgpool-II/pgpool.conf root@server3:/etc/pgpool-II/pgpool.conf

11、拷贝脚本文件并修改

[all servers]# cp -p /etc/pgpool-II/escalation.sh{.sample,}

[all servers]# cp -p /etc/pgpool-II/failover.sh{.sample,}

[all servers]# cp -p /etc/pgpool-II/follow_primary.sh{.sample,}

根据需要修改脚本VIP及服务器网卡名相关目录等变量

[all servers]#chown postgres:postgres /etc/pgpool-II/escalation.sh

[all servers]#chown postgres:postgres /etc/pgpool-II/{failover.sh,follow_primary.sh}

12、在线恢复配置

[server01]# cp -p /etc/pgpool-II/recovery_1st_stage.sample /mnt/postgresql/data/recovery_1st_stage [server01]# cp -p /etc/pgpool-II/pgpool_remote_start.sample /mnt/postgresql/data/pgpool_remote_start

[server01]# chown postgres:postgres /mnt/postgresql/data/{recovery_1st_stage,pgpool_remote_start}

修改PGHOME

PGHOME=/usr/pgsql-10

[server01]#vim /mnt/postgresql/data/recovery_1st_stage

[server01]#vim /mnt/postgresql/data/pgpool_remote_start

13、按序启动postgresql及pgpool服务

[all servers]#systemctl start postgresql-10

[all servers]#systemctl start pgpool

14、验证主从同步

[server01]#su - postgres

[server01]$psql -U postgres

#select client_addr,usename,backend_start,application_name,sync_state,sync_priority FROM pg_stat_replication;

15、验证pgpoll读写分离及状态查询

[server01]$psql -h 10.88.66.228 -p 9999 -U postgres

#show pool_nodes;

#show pool_backend_stats;

16、参考文献：

Pgpool-II + Watchdog Setup Example ###pgpoll 官方文档

PostgreSQL高可用中间件—Pgpool-Ⅱ__雪辉_的博客-CSDN博客_postgresql 中间件

Postgresql + Pgpool 主从及高可用配置_zwk0213的博客-CSDN博客_pgpool

PostgreSQL-10主从物理流复制_慕禾的博客-CSDN博客_postgresql10 流复制 ###主从复制