oracle11g断电后无法启动,oracle11g RAC 启动时无法识别ASM

环境：RHEL5.5 + oracle11g RAC

客户联系说关闭cluster后，重启启动，发现CRS无法启动。提示Cannot communicate with Cluster Ready Services。

登录主机检查

[root@rac-2 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

检查RAC的日志

[grid@rac-2 rac-2]$ tail -100 alertrac-2.log | more

2014-01-23 03:16:17.396

[ohasd(3899)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac-2'.

2014-01-23 03:16:18.697

[crsd(22676)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u02/11.2.0/grid/log/rac-2/crsd/crsd.log.

2014-01-23 03:16:18.704

[crsd(22676)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

ORA-15077: could not locate ASM instance serving a required diskgroup

]. Details at (:CRSD00111:) in /u02/11.2.0/grid/log/rac-2/crsd/crsd.log.

2014-01-23 03:16:19.433

[ohasd(3899)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac-2'.

2014-01-23 03:16:20.737

[crsd(22685)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u02/11.2.0/grid/log/rac-2/crsd/crsd.log.

2014-01-23 03:16:20.747

[crsd(22685)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

ORA-15077: could not locate ASM instance serving a required diskgroup

]. Details at (:CRSD00111:) in /u02/11.2.0/grid/log/rac-2/crsd/crsd.log.2014-01-23 03:16:21.473

[ohasd(3899)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac-2'.

2014-01-23 03:16:21.473

[ohasd(3899)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

检查crsd.log

2014-01-23 03:16:20.461: [ CRSMAIN][1106286912] Policy Engine is not initialized yet!

2014-01-23 03:16:20.463: [ CRSMAIN][3556262304] Initializing OCR

[   CLWAL][3556262304]clsw_Initialize: OLR initlevel [70000]

2014-01-23 03:16:20.735: [  OCRASM][3556262304]proprasmo: Error in open/create file in dg [ORC_VOTE]

[  OCRASM][3556262304]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

2014-01-23 03:16:20.735: [  OCRASM][3556262304]ASM Error Stack : ORA-15077: could not locate ASM instance serving a required diskgroup

2014-01-23 03:16:20.737: [  OCRASM][3556262304]proprasmo: kgfoCheckMount returned [7]

2014-01-23 03:16:20.737: [  OCRASM][3556262304]proprasmo: The ASM instance is down

2014-01-23 03:16:20.738: [  OCRRAW][3556262304]proprioo: Failed to open [+ORC_VOTE]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.

2014-01-23 03:16:20.738: [  OCRRAW][3556262304]proprioo: No OCR/OLR devices are usable2014-01-23 03:16:20.738: [  OCRASM][3556262304]proprasmcl: asmhandle is NULL

2014-01-23 03:16:20.738: [    GIPC][3556262304] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 690], original from [clsss.c : 5326]

2014-01-23 03:16:20.740: [ default][3556262304]clsvactversion:4: Retrieving Active Version from local storage.

2014-01-23 03:16:20.742: [  OCRRAW][3556262304]proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.

2014-01-23 03:16:20.745: [  OCRRAW][3556262304]proprinit: Could not open raw device

2014-01-23 03:16:20.745: [  OCRASM][3556262304]proprasmcl: asmhandle is NULL

2014-01-23 03:16:20.746: [  OCRAPI][3556262304]a_init:16!: Backend init unsuccessful : [26]

2014-01-23 03:16:20.747: [  CRSOCR][3556262304] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage

ORA-15077: could not locate ASM instance serving a required diskgroup

2014-01-23 03:16:20.748: [ CRSMAIN][3556262304] Created alert : (:CRSD00111:) :  Could not init OCR, error: PROC-26: Error while accessing the physical storage

ORA-15077: could not locate ASM instance serving a required diskgroup

2014-01-23 03:16:20.748: [    CRSD][3556262304][PANIC] CRSD exiting: Could not init OCR, code: 26

2014-01-23 03:16:20.748: [    CRSD][3556262304] Done.

从错误信息判断是ASM出现了问题，检查ASM磁盘

[root@rac-2 ~]# /etc/init.d/oracleasm listdisks

ASMDATA01

ASMDATA02

ASMDATA03

OCR_VOTE

磁盘是存在的。

关闭CRS后，检查CRS相关进程

[root@rac-2 ~]# ps -ef | grep d.bin

root      3899     1  0 Jan13 ?        00:18:59 /u02/11.2.0/grid/bin/ohasd.bin reboot

grid      4267     1  0 Jan13 ?        00:34:32 /u02/11.2.0/grid/bin/oraagent.bin

grid      4280     1  0 Jan13 ?        00:00:16 /u02/11.2.0/grid/bin/mdnsd.bin

grid      4293     1  0 Jan13 ?        00:06:10 /u02/11.2.0/grid/bin/gpnpd.bin

root      4304     1  0 Jan13 ?        01:31:25 /u02/11.2.0/grid/bin/orarootagent.bin

grid      4307     1  0 Jan13 ?        00:27:27 /u02/11.2.0/grid/bin/gipcd.bin

root      4322     1  0 Jan13 ?        00:45:33 /u02/11.2.0/grid/bin/osysmond.bin

root      4332     1  0 Jan13 ?        00:01:24 /u02/11.2.0/grid/bin/cssdmonitor

root      4350     1  0 Jan13 ?        00:02:39 /u02/11.2.0/grid/bin/cssdagent

grid      4362     1  0 Jan13 ?        01:45:38 /u02/11.2.0/grid/bin/ocssd.bin

root      4437     1  0 Jan13 ?        00:28:42 /u02/11.2.0/grid/bin/octssd.bin reboot

grid      4461     1  0 Jan13 ?        00:00:22 /u02/11.2.0/grid/bin/evmd.bin

grid      4843  4461  0 Jan13 ?        00:00:00 /u02/11.2.0/grid/bin/evmlogger.bin -o /u02/11.2.0/grid/evm/log/evmlogger.info -l /u02/11.2.0/grid/evm/log/evmlogger.log

root      4941     1  0 Jan13 ?        00:21:18 /u02/11.2.0/grid/bin/ologgerd -m rac-1 -r -d /u02/11.2.0/grid/crf/db/rac-2

root     23122 22979  0 03:54 pts/3    00:00:00 grep d.bin

CRS已经关闭，但是好多进程没有释放。手动kill掉这些进程

[root@rac-2 ~]# ps -ef | grep d.bin | awk '{print $2}' | xargs kill -9

kill 23131: No such process

重启CRS，问题解决。