Greenplum集群状态检查及失败segment修复(上)

最新推荐文章于 2024-05-31 14:54:28 发布
最新推荐文章于 2024-05-31 14:54:28 发布
阅读量2.9k 收藏 4
点赞数
文章标签： 大数据 数据库 python
原文链接：https://my.oschina.net/levinehuang/blog/739626
版权

2019独角兽企业重金招聘Python工程师标准>>> hot3.png

一、GP集群运行状态检查

  1. 查gp_segment_configuration，status='d'表示segment失败，status='u'表示segment正常
SELECT * FROM gp_segment_configuration WHERE status='d';
  1. gpstate查看各个primary、mirror segment状态信息 集群状态：
    9个 primary segment失败
    3个 mirror segment失败
    此时，GP集群增删改查均正常，猜测是mirror segment起了作用。
[gpadmin@gpm ~]$ gpstate
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: 
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May  9 2016 22:28:40'
20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Gathering data from segments...
. 
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-Greenplum instance status summary
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Master instance                                           = Active
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Master standby                                            = gps1
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Standby master state                                      = Standby host passive
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total segment instance count from metadata                = 24
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Primary Segment Status
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total primary segments                                    = 12
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total primary segment valid (at master)                   = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total primary segment failures (at master)                = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number postmaster processes found                   = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Mirror Segment Status
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total mirror segments                                     = 12
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total mirror segment valid (at master)                    = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total mirror segment failures (at master)                 = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 3                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number postmaster processes found                   = 9
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments   = 9                      <<<<<<<<
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-   Total number mirror segments acting as mirror segments    = 3
20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
  1. gpstate -m列出mirror 实例的状态和配置信息。 失败的primary segment被mirror segment顶起来了，可以看到，有9个mirror segment Acting as Primary。
[gpadmin@gpm ~]$ gpstate -m
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May  9 2016 22:28:40'
20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Type = Group
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   Mirror   Datadir                             Port    Status              Data Status       
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data5/gpdata/mirror/gpseg0    50000   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data6/gpdata/mirror/gpseg1    50001   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data7/gpdata/mirror/gpseg2    50002   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps2     /hdfs/data8/gpdata/mirror/gpseg3    50003   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data5/gpdata/mirror/gpseg4    50000   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data6/gpdata/mirror/gpseg5    50001   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data7/gpdata/mirror/gpseg6    50002   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps3     /hdfs/data8/gpdata/mirror/gpseg7    50003   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1     /hdfs/data5/gpdata/mirror/gpseg8    50000   Failed                                <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1     /hdfs/data6/gpdata/mirror/gpseg9    50001   Failed                                <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-   gps1     /hdfs/data7/gpdata/mirror/gpseg10   50002   Acting as Primary   Change Tracking
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1     /hdfs/data8/gpdata/mirror/gpseg11   50003   Failed                                <<<<<<<<
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 segment(s) configured as mirror(s) are acting as primaries
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-3 segment(s) configured as mirror(s) have failed
20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 mirror segment(s) acting as primaries are in change tracking
  1. gpstate -c 查看primary instance 和 mirror instance 的对应关系
[gpadmin@gpm ~]$ gpstate -c
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -c
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1'
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May  9 2016 22:28:40'
20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master...
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Type = Group
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:-   Status                          Data State        Primary   Datadir                              Port    Mirror   Datadir                             Port
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data1/gpdata/primary/gpseg0    40000   gps2     /hdfs/data5/gpdata/mirror/gpseg0    50000   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data2/gpdata/primary/gpseg1    40001   gps2     /hdfs/data6/gpdata/mirror/gpseg1    50001   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data3/gpdata/primary/gpseg2    40002   gps2     /hdfs/data7/gpdata/mirror/gpseg2    50002   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps1      /hdfs/data4/gpdata/primary/gpseg3    40003   gps2     /hdfs/data8/gpdata/mirror/gpseg3    50003   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data1/gpdata/primary/gpseg4    40000   gps3     /hdfs/data5/gpdata/mirror/gpseg4    50000   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data2/gpdata/primary/gpseg5    40001   gps3     /hdfs/data6/gpdata/mirror/gpseg5    50001   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data3/gpdata/primary/gpseg6    40002   gps3     /hdfs/data7/gpdata/mirror/gpseg6    50002   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps2      /hdfs/data4/gpdata/primary/gpseg7    40003   gps3     /hdfs/data8/gpdata/mirror/gpseg7    50003   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed   Change Tracking   gps3      /hdfs/data1/gpdata/primary/gpseg8    40000   gps1     /hdfs/data5/gpdata/mirror/gpseg8    50000   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed   Change Tracking   gps3      /hdfs/data2/gpdata/primary/gpseg9    40001   gps1     /hdfs/data6/gpdata/mirror/gpseg9    50001   <<<<<<<<
20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed   Change Tracking   gps3      /hdfs/data3/gpdata/primary/gpseg10   40002   gps1     /hdfs/data7/gpdata/mirror/gpseg10   50002   <<
最低0.47元/天 解锁文章
weixin_34014277
关注
  • 0
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Greenplum4.3.9集群部署.docx
07-30
本文是在Linux上，以CentOS7.6为操作系统版本，详细讲解GreemPlum（即GP）数据的的集群搭建
Greenplum初探-数据哈希分布与集群扩容
02-24
Greenplum（GP）是分布式数据库，因此，数据的分布是基础。GP提供了多种分布策略：哈希分布、随机分布和复制表。其中，最常用的就是哈希分布。本篇文章我将向大家介绍GP的哈希分布。首先，我们先回顾一下上篇文章用于调试的那张表：大家可以看到建表语句末尾有DISTRIBUTEDBY(c1)，这就表示上面这张表是一张哈希分布表，且通过列c1的值散列数据。我们再来看下表里的数据：因为psql默认连接的是Master，所以这里只能看到整张表的数据。如果我们想观察数据在Segment中的分布情况，那么有没有办法只看某个Segment中存储的数据呢？答案是有的。由于Segment也是一个Postgres
greenplum-记录一个gp数据库segment节点双宕机问题暴力解决方式（不推荐）
qq_38195893的博客
04-03 501
首先存在一个或多个segment的primary节点掉线,致使该segment的mirror节点启动.在mirror节点启动后,在该mirror节点对应的pg_xlog文件夹下,会一直记录系统日志,以保证在该segment的primary节点启动后的数据一致性.
Greenplum 集群数据库部署
06-09
RHEL6.5 平台 Greenplum 4.3 集群数据库部署
数据库GreenPlum集群环境配置及搭建.doc
10-06
数据库GreenPlum集群环境配置及搭建.doc
Greenplum不同集群之间同步数据方法及性能1
08-04
2.1 查看原始表的大小行数与结构 2 2.2 同步语句2 2.3 查看 cpu 与内存的使用情况3 2.3.1 查看 Master CPU 与内存使用情况3
查看GP各segment信息
mengbing1990的博客
07-10 187
SELECT * FROM gp_segment_configuration order by hostname,role asc;
Greenplum数据库故障分析——UDP Packet Lost（packet reassembles failed）
肥叔菌的博客
07-05 1448
最近在项目中遇到SQL挂死，业务无法继续进行下去的情况。在经过几次测试验证（将挂死主机节点踢出集群、将udpifc切换为tcp），该问题会消失。将挂死主机节点加入集群会直接复现。但是将udp切换为tcp后，经过一段时间再切换为udp后，不会复现，但是经过一段时间后，该问题又会复现。 经过排查后，定位在于udp无法成功组包，主要表现在某台主机packet reassembles failed增长量和速度远高于其他主机。 一. Packet reassemble 当发送的UDP报文长度大于MTU大小时，IP层会
Greenplum failed segment的恢复方法
皓月如我的专栏
02-02 5591
转载自：li0924的博客《Greenplum failed segment的恢复方法》 【前记】 Segment检测及故障切换机制 GP Master首先会检测Primary状态，如果Primary不可连通，那么将会检测Mirror状态，Primary/Mirror状态总共有4种： 1. Primary活着，Mirror活着。GP Master探测Primary成功之后直接返回，进行下一
故障分析 | Greenplum Segment 故障处理
ActionTech的博客
02-02 822
作者：杨文DBA，负责客户项目的需求与维护，会点数据库，不限于MySQL、Redis、Cassandra、GreenPlum、ClickHouse、Elastic、TDSQL等等。本文来源：原创投稿*爱可生开源社区出品，原创内容未经授权不得随意使用，转载请联系小编并注明来源。
Greenplum数据库中segment的故障检测与恢复
jmx_bigdata的博客
11-28 5310
1.Greenplum数据库中segment故障检测 1.1概述 Greenplum数据库服务器（Postgres）有一个子进程，该子进程为ftsprobe，主要作用是处理故障检测。 ftsprobe 监视Greenplum数据库阵列，它以可以配置的间隔连接并扫描所有segment和数据库进程。 如果 ftsprobe无法连接到segment，它会在Greenplum数据库系统目录中将seg...
greenplum一次节点迁移
wxc20062006的专栏
11-24 1751
greenplum节点迁移
greenplum(gp)增加mirror镜像配置
wt334502157的博客
09-07 1167
gpaddmirrors工具为初始仅配置了主Segment实例的现有Greenplum数据库系统配置镜像 Segment实例。该工具创建镜像实例并开始主Segment实例和镜像Segment实例之间的在线复制进程。一旦所有 的镜像与其主Segment同步好，用户的Greenplum数据库系统就建立了完全的数据冗余。 Important: 在在线复制进程中，Greenplum数据库应处于静止状态，负载和其他查询不应该运行。
greenplum常见问题及解决方法
热门推荐
q936889811的博客
01-02 4万+
文章目录1、错误：数据库初始化：gpinitsystem -c gpconfigs/gpinitsystem_config -h list2、错误 ：执行检查：gpcheck -f list3、错误：gpadmin-[CRITICAL]:-gpstate failed. (Reason='Environment Variable MASTER_DATA_DIRECTORY not set!') e...
MPP install
weixin_30408309的博客
07-18 681
greenplum install 注意事项：1.hostname 不能大写##2.如果segment和mirror副本数比较大，建议调整/etc/ssh/sshd_config的maxconnect数值3.系统参数需要修改kernel.sem = 500 1024000 200 8192 ##############这个配置是三个节点 6块磁盘，每个磁盘6个segment 6个mi...
处理大数据的关键技术？
喜欢猪猪
05-30 1068
处理大数据涉及多种技术和工具，这些技术和工具各有其专长，能够解决大数据处理中的不同问题。
深入解析CRM客户关系系统：技术架构与功能实现的融合创新
最新发布
m0_65038841的博客
05-31 424
通过以上技术架构和功能实现，CRM客户关系管理系统能够帮助企业实现客户信息的集中管理、销售过程的自动化、营销活动的精准投放以及客户服务的提升，从而提高企业的市场竞争力和客户满意度。容器化与微服务：采用Docker和Kubernetes等技术，实现系统的容器化和微服务架构，提高系统可扩展性和部署灵活性。营销活动管理：创建、发布和管理各种营销活动，如优惠券、促销活动和线上线下活动等，支持活动效果分析。客户信息管理：实现对客户基本信息的录入、查询、修改和删除功能，支持多维度客户细分和标签管理。
Apache Impala 4.4.0正式发布了！
Stiga Huang的专栏
05-28 971
Apache Impala 4.4.0正式发布了！
greenplum集群部署
06-13
Greenplum 是一个高性能、可扩展的关系型数据库管理系统，它是基于 PostgreSQL 构建的。以下是 Greenplum 集群部署的大体步骤： 1. 确定集群规模和硬件配置：根据需求确定 Greenplum 集群的规模和硬件配置，包括节点数、主机配置、磁盘空间、网络带宽等。 2. 安装依赖软件包：安装 Greenplum 需要的依赖软件包，包括 SSH、Python、Java 等。 3. 安装 Greenplum 数据库软件：下载 Greenplum 数据库软件并安装到每个节点上。 4. 配置 Greenplum 集群：在每个节点上配置 Greenplum 集群的基本信息，包括主机名、IP 地址、端口号等。 5. 初始化 Greenplum 集群：在任意一个节点上执行 Greenplum 集群初始化命令，该命令会创建数据库和初始化数据库模板。 6. 创建 Greenplum 集群用户：创建 Greenplum 集群用户，并授权给该用户访问数据库。 7. 启动 Greenplum 集群：在任意一个节点上启动 Greenplum 集群，并检查各个节点的状态。 8. 创建和管理数据库：使用 Greenplum 命令行工具或其他客户端工具创建和管理数据库，包括创建表、导入数据、执行查询等。 以上是 Greenplum 集群部署的大体步骤，具体步骤和命令可以参考 Greenplum 官方文档。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值