crsd.log报错: CAAMonitorHandler :: 0:Could not join /oracle/product/cr

一、环境描述:

    AIX 5.3 + ORACLE 10.2.0.4 RAC

二、问题描述:

今天给一套数据库做巡检是发现CRS日志crsd.log频繁报错vip error,以下截取部分日志:

<p> </p><p>2014-08-22 17:03:03.497: [  CRSEVT][11205]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)
2014-08-22 17:03:03.498: [  CRSAPP][11205]32CheckResource error for ora.p520a.vip error code = -2
<strong><span style="color:#ff0000;">2014-08-24 17:02:53.137: [  CRSEVT][11072]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</span></strong></p><p>2014-08-24 17:02:54.229: [  CRSEVT][11072]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)
2014-08-24 17:02:54.229: [  CRSAPP][11072]32CheckResource error for ora.p520a.vip error code = -2
2014-08-26 17:02:51.141: [  CRSEVT][11199]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p><p>2014-08-26 17:02:52.131: [  CRSEVT][11199]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)
2014-08-26 17:02:52.132: [  CRSAPP][11199]32CheckResource error for ora.p520a.vip error code = -2
2014-08-27 17:03:05.161: [  CRSEVT][11259]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p><p>2014-08-27 17:03:06.859: [  CRSEVT][11259]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)
2014-08-27 17:03:06.859: [  CRSAPP][11259]32CheckResource error for ora.p520a.vip error code = -2
2014-08-28 17:03:37.069: [  CRSEVT][11067]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p><p>2014-08-28 17:03:40.451: [  CRSEVT][11067]32CAAMonitorHandler :: 0:Action Script /oracle/product/crs/bin/racgwrap(check) timed out for ora.p520a.vip! (timeout=60)
2014-08-28 17:03:40.451: [  CRSAPP][11067]32CheckResource error for ora.p520a.vip error code = -2
2014-09-01 17:03:08.068: [  CRSEVT][11061]32CAAMonitorHandler :: 0:Could not join /oracle/product/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child</p>

三、解决方法

通过METALINK搜索，以上报错信息与BUG 8572205症状相吻合，不一定需要打补丁也可以通过修改配置文件来规避

该BUG的现象是,BUG发作的时间很短,但如果没有配置FAF,会话将中止而且新会话无法连接p520a.vip

BUG的详细描述如下：

===================================================================================

Hdr: 8572205 10.2.0.4 PCW 10.2.0.4 RACG PRODID-5 PORTID-23
Abstract: CHILDCRASH, OS ERROR: 0, OTHER: ABNORMAL TERMINATION OF CHILD
 
*** 06/03/09 10:03 am ***
TAR:
----
7537479.993
 
PROBLEM:
--------
complete outage in the 4 instances out of 6 because of the following issue:
=====
    
    2009-06-02 02:26:00.470: [  CRSEVT][911170] CAAMonitorHandler :: 0:Could 
not join
    /opt/oracle/product/10.2.0/crs/bin/racgwrap(check)
    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 
0, other: Abnormal
    termination of the child
    
    2009-06-02 02:26:00.470: [  CRSEVT][911170] CAAMonitorHandler :: 0:Action 
Script
    /opt/oracle/product/10.2.0/crs/bin/racgwrap(check) timed out for 
ora.eprvd4244.vip! (timeout=60)
    2009-06-02 02:26:00.470: [  CRSAPP][911170] CheckResource error for 
ora.eprvd4244.vip error code =
    -2
    2009-06-02 02:35:22.561: [  CRSEVT][911158] CAAMonitorHandler :: 0:Could 
not join
    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check)
    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 
0, other: Abnormal
    termination of the child
    
    2009-06-02 02:35:22.561: [  CRSEVT][911158] CAAMonitorHandler :: 0:Action 
Script
    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check) timed out for 
ora.se001p.se001p5.inst!
    (timeout=600)
    2009-06-02 02:35:22.561: [  CRSAPP][911158] CheckResource error for 
ora.se001p.se001p5.inst error
    code = -2
    2009-06-02 02:35:23.101: [  CRSEVT][911159] CAAMonitorHandler :: 0:Could 
not join
    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check)
    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 
0, other: Abnormal
    termination of the child
    
    2009-06-02 02:35:23.101: [  CRSEVT][911159] CAAMonitorHandler :: 0:Action 
Script
    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check) timed out for
    ora.eprvd4244.LISTENER_OFAC0P_EPRVD4244.lsnr! (timeout=600)
    2009-06-02 02:35:23.101: [  CRSAPP][911159] CheckResource error for
    ora.eprvd4244.LISTENER_OFAC0P_EPRVD4244.lsnr error code = -2
    2009-06-02 02:35:23.691: [  CRSEVT][911162] CAAMonitorHandler :: 0:Could 
not join
    /opt/oracle/product/10.2.0/racdb_04/bin/racgwrap(check)
    category: 1234, operation: scls_process_join, loc: childcrash, OS error: 
0, other: Abnormal
    termination of the child
=======
 
DIAGNOSTIC ANALYSIS:
--------------------
This issue is already addressed in bug:6196746.  This bug is fixed in 
10.2.0.5. and
    11.1.0.7.The Workaround is as follows:
    =====
    
    1. Stop CRS on the Node.
    
    2. Make a copy of racgwrap located under $ORACLE_HOME/bin and 
$CRS_HOME/bin on the Node
    
    3. Edit the file racgwrap and modify the last 3 lines from:
    
    $ORACLE_HOME/bin/racgmain "$"
    status=$?
    exit $status
    
    to:
    
    # Line added to test fix for Bug 6196746
    exec $ORACLE_HOME/bin/racgmain "$"
    
    4. Restart CRS and make sure that all the resources are starts.
    =====
 
WORKAROUND:
-----------
The Workaround is NOT working in rolling way. customer CAN NOT have the 
complete outage in the cluster as its their vital business generating system
 
RELATED BUGS:
-------------
Bug 6196746 - HUGE AND GROWING LIST OF RACG CHECK VIP PROCESSES, TIMEOUT 

===================================================================================

 

-------------------------------------------------------------------------------------------------

本文来自于我的技术博客 http://blog.csdn.net/robo23

转载请标注源文链接,否则追究法律责任!