11g rac terminating the instance due to error 481 节点驱逐

版本

       aix 6.1 

       oracle 11.2.0.3.0 rac

问题描述

     2节点rac，其中一个节点无法启动，startup nomount无法成功，报错ORA-03113: end-of-file on communication channel

    alert中发现如下内容

    PMON (ospid: nnnn): terminating the instance due to error 481

这里把文档Doc ID 1383737.1贴出来，做下记录

 
 ASM on Non First Node (Second or Other Node) Fails to Come up With: PMON (ospid: nnnn): terminating the instance due to error 481 (Doc ID 1383737.1)
 
Applies to:

   Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
   Information in this document applies to any platform.

Purpose

This note lists common causes of ASM start up failure with the following error on non-first node (second or others):

  alert_<ASMn>.log from non-first node
  lmon registered with NM - instance number 2 (internal mem no 1)
  Tue Dec 06 06:16:15 2011
 System state dump requested by (instance=2, osid=19095 (PMON)), summary=[abnormal instance termination].
 System State dumped to trace file /g01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_19138.trc
 Tue Dec 06 06:16:15 2011
 PMON (ospid: 19095): terminating the instance due to error 481
 Dumping diagnostic data in directory=[cdmp_20111206061615], requested by (instance=2, osid=19095 (PMON)), summary=[abnormal instance termination].
 Tue Dec 06 06:16:15 2011
 ORA-1092 : opitsk aborting process

Note: ASM instance terminates shortly after "lmon registered with NM"

If ASM on non-first node was running previously, likely the following will be in alert.log when it failed originally:

 ..
 IPC Send timeout detected. Sender: ospid 32231 [oracle@ftdcslsedw01b (PING)]
 ..
 ORA-29740: evicted by instance number 1, group incarnation 10
 ..

 

 diag trace from non-first ASM (+ASMn_diag_<pid>.trc)
 kjzdattdlm: Can not attach to DLM