Recovery (bring GPDB up) from "FATAL:Global sequence number 1131954 less than maximum value 1131958

"Recovery (bring GPDB up) from "FATAL:Global sequence number 1131954 less than maximum value 1131958 found in scan ('gp_persistent_relation_node')""

GPDB is down due to crash recovery failure with the following error message:

2012-03-06 02:35:30.275573 CST,,,p20291,th-101075200,,,,0,,,seg-1,,,,,"FATAL","XX000","Global sequence number 1131954 less than maxi
mum value 1131958 found in scan ('gp_persistent_relation_node')(cdbpersistentstore.c:487)",,,,,,,0,,"cdbpersistentstore.c",487,"Sta
ck trace:
1 0xa3338a postgres <symbol not found> (elog.c:454)
2 0xa350f8 postgres elog_finish (elog.c:1365)
3 0xb7ad2e postgres PersistentStore_InitScanUnderLock (cdbpersistentstore.c:487)
4 0xb6587e postgres PersistentFileSysObj_StartupInitScan (cdbpersistentfilesysobj.c:642)
5 0x508775 postgres StartupXLOG (xlog.c:6891)
6 0x510aed postgres StartupProcessMain (xlog.c:10341)
7 0x584d05 postgres AuxiliaryProcessMain (bootstrap.c:452)
8 0x84b774 postgres <symbol not found> (postmaster.c:7062)
9 0x84b95a postgres StartMasterOrPrimaryPostmasterProcesses (postmaster.c:1558)
10 0x85af20 postgres doRequestedPrimaryMirrorModeTransitions (primary_mirror_mode.c:1631)
11 0x8560e0 postgres PostmasterMain (postmaster.c:2108)
12 0x76530a postgres main (main.c:212)
13 0x7f7cf8649bc6 libc.so.6 __libc_start_main (??:0)
14 0x475089 postgres <symbol not found> (??:0)

A solution exists for this issue, but intervention from EMC technical support personnel is required. Contact the EMC Customer Support Center, or your service representative for technical assistance and quote this solution ID (emc293101).

Before taking any action, please contact the experienced or senior Greenplum support engineer to assist you in implementing the fix.

A new GUC "gp_persistent_repair_global_sequence" has been introduced to use the scanned maximum value as the current global sequence number. As a result, the GPDB is brought up. No catalog inconsistency is introduced with the recovery procedure.

Set this parameter on every segment that experiences this error:

gp_crash_recovery_abort_suppress_fatal=on

Immediately after starting the database, it is strongly advised to run gpcheckcat to find and fix the inconsistency. Please look for assistance in interpreting and fixing the issues reported in the gpcheckcat output.