gp数据库主节点故障导致数据库无法重启报错

由于机房电源割接失误导致所有服务器全部挂掉，导致数据库故障，重启数据库一直报错，以下是解决方法希望可以帮到有同样问题的小伙伴。

故障原因：Greenplum 由于备master节点故障导致gp无法启动处理

1、已经配置了gpstandby节点，如果gpstandby故障是不会影响Greenplum正常运行，但是如果此时关闭master主节点则无法启动需要将原gpstandby节点移除。

1、启动master节点失败

[gpadmin@gpmaster ~]$ gpstart
20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Starting gpstart with args:
20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Gathering information and validating the environment…
20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Binary Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’
20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Catalog Version: ‘201310150’
20200908:11:30:38:025343 gpstart:gpmaster:gpadmin-[INFO]:-Starting Master instance in admin mode
20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master…
20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Setting new master era
20200908:11:30:39:025343 gpstart:gpmaster:gpadmin-[INFO]:-Master Started…
20200908:11:30:40:025343 gpstart:gpmaster:gpadmin-[CRITICAL]:-Error occurred: non-zero rc: 2
Command was: ‘ssh -o ‘StrictHostKeyChecking no’ gpstandby “. /usr/local/gpdb-5.0.0/greenplum_path.sh; $GPHOME/bin/pg_controldata /master/gpseg-1”’
rc=2, stdout=’’, stderr='pg_controldata: could not open file “/master/gpseg-1/global/pg_control” for reading: No such file or directory
’

2、移除问题gpstandby

[gpadmin@gpmaster ~]$ gpinitstandby -r gpstandby
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Warm master standby removal parameters
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master hostname = gpmaster
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master data directory = /master/gpseg-1
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master port = 5432
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master hostname = gpstandby
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master port = 5432
20200908:11:30:53:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master data directory = /master/gpseg-1
Do you want to continue with deleting the standby master? Yy|Nn (default=N):

y
20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Removing standby master from catalog…
20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Database catalog updated successfully.
20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Removing filespace directories on standby master…
20200908:11:30:56:025380 gpinitstandby:gpmaster:gpadmin-[INFO]:-Successfully removed standby master

3、停止master节点
[gpadmin@gpmaster ~]$ gpstop
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Starting gpstop with args:
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Gathering information and validating the environment…
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master…
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Greenplum Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:---------------------------------------------
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Master instance parameters
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:---------------------------------------------
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Master Greenplum instance process active PID = 19445
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Database = template1
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Master port = 5432
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Master directory = /master/gpseg-1
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Shutdown mode = smart
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Timeout = 120
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Shutdown Master standby host = Off
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:---------------------------------------------
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:-Segment instances that will be shutdown:
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:---------------------------------------------
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- Host Datadir Port Status
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata50/gpseg0 40000 u
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata/mirror50/gpseg0 41000 u
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata50/gpseg1 40000 u
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata/mirror50/gpseg1 41000 u
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata50/gpseg2 40000 u
20200908:11:31:11:025413 gpstop:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata/mirror50/gpseg2 41000 u

Continue with Greenplum instance shutdown Yy|Nn (default=N):

y
20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-There are 0 connections to the database
20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=‘smart’
20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Master host=gpmaster
20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart
20200908:11:31:04:025413 gpstop:gpmaster:gpadmin-[INFO]:-Master segment instance directory=/master/gpseg-1
20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-Terminating processes for segment /master/gpseg-1
20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-No standby master host configured
20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait…
20200908:11:31:05:025413 gpstop:gpmaster:gpadmin-[INFO]:-0.00% of jobs completed
20200908:11:31:15:025413 gpstop:gpmaster:gpadmin-[INFO]:-100.00% of jobs completed
20200908:11:31:15:025413 gpstop:gpmaster:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait…
20200908:11:31:15:025413 gpstop:gpmaster:gpadmin-[INFO]:-0.00% of jobs completed
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-100.00% of jobs completed
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:- Segments stopped successfully = 6
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:- Segments with errors during stop = 0
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Successfully shutdown 6 of 6 segment instances
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-No leftover gpmmon process found
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20200908:11:31:25:025413 gpstop:gpmaster:gpadmin-[INFO]:-Cleaning up leftover shared memory

4、重新启动master
[gpadmin@gpmaster ~]$ gpstart
20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Starting gpstart with args:
20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Gathering information and validating the environment…
20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Binary Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’
20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Greenplum Catalog Version: ‘201310150’
20200908:11:31:34:025522 gpstart:gpmaster:gpadmin-[INFO]:-Starting Master instance in admin mode
20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master…
20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Setting new master era
20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master Started…
20200908:11:31:35:025522 gpstart:gpmaster:gpadmin-[INFO]:-Shutting down master
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:---------------------------
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master instance parameters
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:---------------------------
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Database = template1
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master Port = 5432
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master directory = /master/gpseg-1
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Timeout = 600 seconds
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Master standby = Off
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:---------------------------------------
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:-Segment instances that will be started
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:---------------------------------------
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- Host Datadir Port Role
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata50/gpseg0 40000 Primary
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata/mirror50/gpseg0 41000 Mirror
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw02 /gpdata50/gpseg1 40000 Primary
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata/mirror50/gpseg1 41000 Mirror
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw03 /gpdata50/gpseg2 40000 Primary
20200908:11:31:37:025522 gpstart:gpmaster:gpadmin-[INFO]:- sdw01 /gpdata/mirror50/gpseg2 41000 Mirror

Continue with Greenplum instance startup Yy|Nn (default=N):

y
20200908:11:31:39:025522 gpstart:gpmaster:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait…
…
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-Process results…
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:- Successful segment starts = 6
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:- Failed segment starts = 0
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-Successfully started 6 of 6 segment instances
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-----------------------------------------------------
20200908:11:31:51:025522 gpstart:gpmaster:gpadmin-[INFO]:-Starting Master instance gpmaster directory /master/gpseg-1
20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[INFO]:-Command pg_ctl reports Master gpmaster instance active
20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[WARNING]:-global name ‘testurl’ is not defined
20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[INFO]:-No standby master configured. skipping…
20200908:11:31:52:025522 gpstart:gpmaster:gpadmin-[INFO]:-Check status of database with gpstate utility
[gpadmin@gpmaster ~]$

5、重新恢复maststandby主节点
[gpadmin@gpmaster ~]$ gpstate -f
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-Starting gpstate with args: -f
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 8.3.23 (Greenplum Database 5.0.0-alpha.0+dev.266.g923109b build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) compiled on Mar 28 2020 11:36:47’
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master…
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-Standby master instance not configured
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:–pg_stat_replication
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:-No entries found.
20200908:11:36:47:025717 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------

[gpadmin@gpmaster ~]$ gpinitstandby -s gpstandby
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Validating environment and parameters for standby initialization…
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Checking for filespace directory /master/gpseg-1 on gpstandby
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master initialization parameters
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master hostname = gpmaster
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master data directory = /master/gpseg-1
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum master port = 5432
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master hostname = gpstandby
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master port = 5432
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum standby master data directory = /master/gpseg-1
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Greenplum update system catalog = On
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:- Filespace locations
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:------------------------------------------------------
20200908:11:37:04:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-pg_system -> /master/gpseg-1
Do you want to continue with standby master initialization? Yy|Nn (default=N):

y
20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby
20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[WARNING]:-Syncing of Greenplum Database extensions has failed.
20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[WARNING]:-Please run gppkg --clean after successful standby initialization.
20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Adding standby master to catalog…
20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Database catalog updated successfully.
20200908:11:37:06:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Updating pg_hba.conf file…
20200908:11:37:08:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-pg_hba.conf files updated successfully.
20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Updating filespace flat files…
20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Filespace flat file updated successfully.
20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Starting standby master
20200908:11:37:17:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Checking if standby master is running on host: gpstandby in directory: /master/gpseg-1
20200908:11:37:22:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Cleaning up pg_hba.conf backup files…
20200908:11:37:23:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully.
20200908:11:37:23:025779 gpinitstandby:gpmaster:gpadmin-[INFO]:-Successfully created standby master on gpstandby

[gpadmin@gpmaster ~]$ gpstate -f
20200908:11:37:27:025887 gpstate:gpmaster:gpadmin-[INFO]:-Starting gpstate with args: -f
20200908:11:37:28:025887 gpstate:gpmaster:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 5.0.0-alpha.0+dev.266.g923109b build dev’
20200908:11:37:28:025887 gpstate:gpmaster:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 8.3.23 (Greenplum Database 5.0.0-alpha.0+dev.266.g923109b build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) compiled on Mar 28 2017 20:09:54’
20200908:11:37:28:025887 gpstate:gpmaster:gpadmin-[INFO]:-Obtaining Segment details from master…
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:-Standby master details
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:-----------------------
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby address = gpstandby
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby data directory = /master/gpseg-1
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby port = 5432
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby PID = 2641
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:- Standby status = Standby host passive
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–pg_stat_replication
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–WAL Sender State: streaming
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Sync state: sync
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Sent Location: 0/24000000
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Flush Location: 0/24000000
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:–Replay Location: 0/24000000
20200908:11:37:29:025887 gpstate:gpmaster:gpadmin-[INFO]:--------------------------------------------------------------