多么痛的一次领悟ORA-21561: OID generation failed

环境为 2个节点的11.2.0.4的RAC环境

重启异常

数据库无法启动，出现报错如下

在日志

 /opt/app/grid/11.2.0/log/racoa02/agent/ohasd/oraagent_grid/oraagent_grid.log

 

 

2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} checkCrsStat 2 CLSCRS_STAT ret: 184
2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} CrsCmd::ClscrsCmdData::destroy
2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} CrsCmd::destroy
2018-10-29 23:26:01.841: [ora.evmd][4040279808]{0:0:2} [start] (:CLSN00107:) clsn_agent::start }
2018-10-29 23:26:01.841: [    AGFW][4040279808]{0:0:2} Command: start for resource: ora.evmd 1 1 completed with status: SUCCESS
2018-10-29 23:26:01.841: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_START[ora.evmd 1 1] ID 4098:438
2018-10-29 23:26:01.841: [ora.evmd][4040279808]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:01.841: [ USRTHRD][4040279808]{0:0:2} Thread:[DaemonCheck:evmd]thread constructor
2018-10-29 23:26:01.842: [ USRTHRD][4040279808]{0:0:2} Thread:[DaemonCheck:evmd]start {
2018-10-29 23:26:01.842: [ USRTHRD][4040279808]{0:0:2} Thread:[DaemonCheck:evmd]start }
2018-10-29 23:26:01.842: [ora.evmd][4040279808]{0:0:2} [check] DaemonAgent::check returned 0
2018-10-29 23:26:01.842: [ora.evmd][4040279808]{0:0:2} [check] Deep check returned 0
2018-10-29 23:26:01.842: [    AGFW][4067276544]{0:0:2} ora.evmd 1 1 state changed from: STARTING to: PARTIAL
2018-10-29 23:26:01.843: [    AGFW][4067276544]{0:0:2} Started implicit monitor for [ora.evmd 1 1] interval=30000 delay=30000
2018-10-29 23:26:01.843: [    AGFW][4067276544]{0:0:2} Agent sending last reply for: RESOURCE_START[ora.evmd 1 1] ID 4098:438
2018-10-29 23:26:01.843: [CLSFRAME][4088489760] TM [MultiThread] is changing desired thread # to 5. Current # is 4
2018-10-29 23:26:03.504: [ora.mdnsd][4040279808]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:05.515: [ora.gpnpd][4069377792]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:06.526: [ora.gipcd][2613040896]{0:0:2} [check] clsdmc_respget return: status=0, ecode=0
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] InstAgent::check prev clsagfw_res_status 2 current clsagfw_res_status 2
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop  shutdown mode: 4
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::getConnection 160 ERROR sid +ASM1 pool status 1
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop pool pConnxn 00000000
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] clsnInstConnection::makeConnectStr UsrOraEnv  m_oracleHome /opt/app/grid/11.2.0 Crshome /opt/app/grid/11.2.0
2018-10-29 23:26:06.750: [ora.asm][4063266560]{0:0:2} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/opt/app/grid/11.2.0/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/opt/app/grid/11.2.0,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2018-10-29 23:26:06.751: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop non pool pConnxn 1 d00404c0
2018-10-29 23:26:06.751: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt: server not attached
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] ORA-21561: OID generation failed

2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt (2) Exception OCIException
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstConnection:connect:excp OCIException OCI error 21561
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop: connect1 errcode 21561
2018-10-29 23:26:06.762: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop: connect2 oracleHome /opt/app/grid/11.2.0 oracleSid +ASM1
2018-10-29 23:26:06.763: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt: server not attached
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] ORA-21561: OID generation failed

2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] InstConnection::connectInt (2) Exception OCIException
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] InstConnection:connect:excp OCIException OCI error 21561
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop: connect2 errcode 21561
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] clsnUtils::error Exception type=2 string=
ORA-21561: OID generation failed

2018-10-29 23:26:06.772: [    AGFW][4063266560]{0:0:2} sending status msg [ORA-21561: OID generation failed
] for start for resource: ora.asm 1 1
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection connection count 1
2018-10-29 23:26:06.772: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection sid  +ASM1, InstConnection d00404c0
2018-10-29 23:26:06.772: [ USRTHRD][4063266560]{0:0:2} InstConnection::breakCall pConnxn:d00404c0  DetachLock:00b01580 m_pSvcH:d00eb5f0
2018-10-29 23:26:06.772: [ USRTHRD][4063266560]{0:0:2} InstConnection:~InstConnection: this d00404c0
2018-10-29 23:26:06.773: [    AGFW][4067276544]{0:0:2} Agent sending reply for: RESOURCE_START[ora.asm 1 1] ID 4098:444
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection delete InstConnection 0x7f1fd00404c0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection freed 1
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::stopConnection
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection connection count 0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection freed 0
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::stopConnection sid +ASM1 status  1
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] InstAgent::stop db/asm
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::stopConnection
2018-10-29 23:26:06.774: [ora.asm][4063266560]{0:0:2} [start] ConnectionPool::removeConnection connection count 0
2018-10-29