Greenplum集群状态检查及失败segment修复(上)

本文介绍了如何检查Greenplum集群的状态,包括使用gp_segment_configuration和gpstate工具。当发现segment失败时,通过gprecoverseg工具进行修复,并详细描述了修复过程,包括确保primary和mirror的同步以及恢复segment到初始角色。最后,文中提到了遗留问题,即对不同场景下的故障恢复还需要更深入的理解。
摘要由CSDN通过智能技术生成

一、GP集群运行状态检查

  1. 查gp_segment_configuration,status='d'表示segment失败,status='u'表示segment正常
SELECT * FROM gp_segment_configuration WHERE status='d';
  1. gpstate查看各个primary、mirror segment状态信息 集群状态:
    9个 primary segment失败
    3个 mirror segment失败
    此时,GP集群增删改查均正常,猜测是mirror segment起了作用。
[gpadmin@gpm ~]$ gpstate
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: 
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May  9 2016 22:28:40'
20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Gathering data from segments...
. 
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-Greenplum instance status summary
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Master instance                                           = Active
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Master standby                                            = gps1
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Standby master state                                      = Standby host passive
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total segment instance count from metadata                = 24
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Primary Segment Status
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total primary segments                                    = 12
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total primary segment valid (at master)                   = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total primary segment failures (at master)                = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number postmaster processes found                   = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Mirror Segment Status
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total mirror segments                                     = 12
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total mirror segment valid (at master)                    = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total mirror segment failures (at master)                 = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number postmaster processes found                   = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments   = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number mirror segments acting as mirror segments    = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
  1. gpstate -m列出mirror 实例的状态和配置信息。 失败的primary segment被mirror segment顶起来了,可以看到,有9个mirror segment Acting as Primary。
[gpadmin@gpm ~]$ gpstate -m
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May  9 2016 22:28:40'
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Type = Group
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   Mirror   Datadir                             Port    Status              Data Status       
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data5/gpdata/mirror/gpseg0    50000   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data6/gpdata/mirror/gpseg1    50001   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data7/gpdata/mirror/gpseg2    50002   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data8/gpdata/mirror/gpseg3    50003   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data5/gpdata/mirror/gpseg4    50000   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data6/gpdata/mirror/gpseg5    50001   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data7/gpdata/mirror/gpseg6    50002   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data8/gpdata/mirror/gpseg7    50003   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1     /hdfs/data5/gpdata/mirror/gpseg8    50000   Failed                                <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1     /hdfs/data6/gpdata/mirror/gpseg9    50001   Failed                                <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps1     /hdfs/data7/gpdata/mirror/gpseg10   50002   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1     /hdfs/data8/gpdata/mirror/gpseg11   50003   Failed                                <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 segment(s) configured as mirror(s) are acting as primaries
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-3 segment(s) configured as mirror(s) have failed
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 mirror segment(s) acting as primaries are in change tracking
  1. gpstate -c 查看primary instance 和 mirror instance 的对应关系
[gpadmin@gpm ~]$ gpstate -c
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -c
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May  9 2016 22:28:40'
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Type = Group
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:-   Status                          Data State        Primary   Datadir                              Port    Mirror   Datadir                             Port
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data1/gpdata/primary/gpseg0    40000   gps2     /hdfs/data5/gpdata/mirror/gpseg0    50000   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data2/gpdata/primary/gpseg1    40001   gps2     /hdfs/data6/gpdata/mirror/gpseg1    50001   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data3/gpdata/primary/gpseg2    40002   gps2     /hdfs/data7/gpdata/mirror/gpseg2    50002   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data4/gpdata/primary/gpseg3    40003   gps2     /hdfs/data8/gpdata/mirror/gpseg3    50003   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data1/gpdata/primary/gpseg4    40000   gps3     /hdfs/data5/gpdata/mirror/gpseg4    50000   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data2/gpdata/primary/gpseg5    40001   gps3     /hdfs/data6/gpdata/mirror/gpseg5    50001   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data3/gpdata/primary/gpseg6    40002   gps3     /hdfs/data7/gpdata/mirror/gpseg6    50002   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data4/gpdata/primary/gpseg7    40003   gps3     /hdfs/data8/gpdata/mirror/gpseg7    50003   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed   Change Tracking   gps3      /hdfs/data1/gpdata/primary/gpseg8    40000   gps1     /hdfs/data5/gpdata/mirror/gpseg8    50000   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed   Change Tracking   gps3      /hdfs/data2/gpdata/primary/gpseg9    40001   gps1     /hdfs/data6/gpdata/mirror/gpseg9    50001   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps3      /hdfs/data3/gpdata/primary/gpseg10   40002   gps1     /hdfs/data7/gpdata/mi
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值