一、GP集群运行状态检查
- 查gp_segment_configuration,status='d'表示segment失败,status='u'表示segment正常
SELECT * FROM gp_segment_configuration WHERE status='d';
- gpstate查看各个primary、mirror segment状态信息 集群状态:
9个 primary segment失败
3个 mirror segment失败
此时,GP集群增删改查均正常,猜测是mirror segment起了作用。
[gpadmin@gpm ~]$ gpstate
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args:
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40'
20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Gathering data from segments...
.
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-Greenplum instance status summary
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Master instance = Active
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Master standby = gps1
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Standby master state = Standby host passive
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total segment instance count from metadata = 24
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Primary Segment Status
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total primary segments = 12
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total primary segment valid (at master) = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total primary segment failures (at master) = 9 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing = 9 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files found = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing = 9 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing = 9 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files found = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing = 9 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes found = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Mirror Segment Status
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total mirror segments = 12
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total mirror segment valid (at master) = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total mirror segment failures (at master) = 3 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing = 3 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files found = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing = 3 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing = 3 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files found = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing = 3 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes found = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments = 9 <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
- gpstate -m列出mirror 实例的状态和配置信息。 失败的primary segment被mirror segment顶起来了,可以看到,有9个mirror segment Acting as Primary。
[gpadmin@gpm ~]$ gpstate -m
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40'
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Type = Group
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 Failed <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 Failed <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 Acting as Primary Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 Failed <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 segment(s) configured as mirror(s) are acting as primaries
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-3 segment(s) configured as mirror(s) have failed
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 mirror segment(s) acting as primaries are in change tracking
- gpstate -c 查看primary instance 和 mirror instance 的对应关系
[gpadmin@gpm ~]$ gpstate -c
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -c
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40'
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Type = Group
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:- Status Data State Primary Datadir Port Mirror Datadir Port
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data1/gpdata/primary/gpseg0 40000 gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data2/gpdata/primary/gpseg1 40001 gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data3/gpdata/primary/gpseg2 40002 gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data4/gpdata/primary/gpseg3 40003 gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data1/gpdata/primary/gpseg4 40000 gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data2/gpdata/primary/gpseg5 40001 gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data3/gpdata/primary/gpseg6 40002 gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data4/gpdata/primary/gpseg7 40003 gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed Change Tracking gps3 /hdfs/data1/gpdata/primary/gpseg8 40000 gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed Change Tracking gps3 /hdfs/data2/gpdata/primary/gpseg9 40001 gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps3 /hdfs/data3/gpdata/primary/gpseg10 40002 gps1 /hdfs/data7/gpdata/mi