AIX6.1系统11203RAC节点down机

alert里大量的
Wed Jan 08 16:42:14 2014
IPC Send timeout detected. Sender: ospid 23396530 [oracle@ptms3db1]
Receiver: inst 2 binc 315765730 ospid 29556890

然后ASM里日志
Wed Jan 08 16:42:49 2014
IPC Send timeout detected. Sender: ospid 10158142 [oracle@ptms3db1 (LMON)]
Receiver: inst 2 binc 116716205 ospid 9502824
Communications reconfiguration: instance_number 2
Wed Jan 08 16:43:46 2014
IPC Send timeout detected. Sender: ospid 9175278 [oracle@ptms3db1 (LMS0)]
Receiver: inst 2 binc 116716132 ospid 9895996
IPC Send timeout to 2.1 inc 4 for msg type 34 from opid 11
。。
Wed Jan 08 16:48:23 2014
LMON (ospid: 10158142) waits for event 'CSS operation: data update' for 216 secs.
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lmhb_6946928.trc  (incident=35298):
ORA-29770: global enqueue process LMON (OSID 10158142) is hung for more than 150 seconds
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_35298/+ASM1_lmhb_6946928_i35298.trc

CRS下节点的告警
2014-01-08 16:44:11.893
[cssd(8454194)]CRS-1663:Member kill issued by PID 28967080 for 1 members, group DBOLTP_RAC. Details at (:CSSGM00044:) in /u01/app/11.2.0/grid/log/ptms3db1/cssd/ocssd.log.
2014-01-08 16:44:42.413
[cssd(8454194)]CRS-1607:Node ptms3db2 is being evicted in cluster incarnation 280992666; details at (:CSSNM00007:) in /u01/app/11.2.0/grid/log/ptms3db1/cssd/ocssd.log.
2014-01-08 16:49:29.676
[/u01/app/11.2.0/grid/bin/oraagent.bin(7143460)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/ptms3db1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2014-01-08 16:49:30.987
[/u01/app/11.2.0/grid/bin/oraagent.bin(15728690)]CRS-5011:Check of resource "OLTP_RAC" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/ptms3db1/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2014-01-08 16:49:59.592
[/u01/app/11.2.0/grid/bin/oraagent.bin(7143460)]CRS-5818:Aborted command 'check' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:810} in /u01/app/11.2.0/grid/log/ptms3db1/agent/ohasd/oraagent_grid/oraagent_grid.log.

已经几天了，Oracle support还在查找原因，也转到了bug team

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/628922/viewspace-1069444/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/628922/viewspace-1069444/