Root.sh failed at Failure at final check of Oracle CRS stack 10 问题

      今天在公司和同事一起测试安装oracle 10g rac for aix6.1+asm安装，在装完

crs集群软件，在第一节点运行root.sh时最后报以下错误:

failed at Failure at final check of Oracle CRS stack

 10

查看$ORA_CRS_HOME/log/p520/client下的相关日志报下面错误

[root@p520:/crs/app/oracle/product/crs_1/log/p520/client]#more clsc.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.
2010-04-16 16:55:05.812: [ COMMCRS][1]clsc_connect: (11068c330) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

2010-04-16 16:55:05.813: [ COMMCRS][1]clsc_connect: (11068b6b0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)
)

2010-04-16 16:55:05.813: [ default][1]Terminating clsd session
2010-04-16 18:41:53.148: [ COMMCRS][1]clsc_connect: (1101bf070) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_p520_crs))

2010-04-16 18:41:53.149: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

2010-04-16 18:41:53.149: [ default][1]Terminating clsd session
2010-04-16 17:42:17.881: [ COMMCRS][1]clsc_connect: (1101bf070) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_p520_crs))

2010-04-16 17:42:17.882: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

2010-04-16 17:42:17.882: [ default][1]Terminating clsd session
2010-04-16 17:45:21.416: [ COMMCRS][1]clsc_connect: (1101bf070) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_p520_crs))

2010-04-16 17:45:21.519: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

crsctl check boot无报错

/etc/init.cssd 无报错

crs进程起不来

没办法，只能查metalink,查到有可能是aix gui 工具没配置，建议在图形窗口运行 install_assist 命令，进行修改时间的配置，配置完成后重新启动操作系统，启动crs,发现有evmd进程没起来。

[oracle@p520:/oracle/app/oracle]$ps -ef|grep crs
    root 229546      1   0 17:45:37      -  0:01 /crs/app/oracle/product/crs_1/bin/crsd.bin reboot
  oracle 327874 569510   0 19:00:57  pts/1  0:00 grep crs
  oracle 389122 360648   0 17:45:41      -  0:00 /bin/sh -c ulimit -c unlimited; cd /crs/app/oracle/product/crs_1/log/p520/cssd;  /crs/app/oracle/product/crs_1/bin/ocssd  || exit $?
    root 409806      1   0 17:46:06      -  0:00 /bin/sh /etc/init.crsd run
  oracle 454904 389122   0 17:45:41      -  0:00 /crs/app/oracle/product/crs_1/bin/ocssd.bin
    root 524338 418004   0 17:45:40      -  0:00 /crs/app/oracle/product/crs_1/bin/oprocd run -t 1000 -m 500

运行init.crs start 命令  evmd进程正常启动

[root@p520:/etc]#init.crs start

[root@p520:/etc]#init.crs start
Startup will be queued to init within 30 seconds.
[root@p520:/etc]#ps -ef|grep crs
    root 229546      1   0 17:45:37      -  0:01 /crs/app/oracle/product/crs_1/bin/crsd.bin reboot
  oracle 389122 360648   0 17:45:41      -  0:00 /bin/sh -c ulimit -c unlimited; cd /crs/app/oracle/product/crs_1/log/p520/cssd;  /crs/app/oracle/product/crs_1/bin/ocssd  || exit $?
    root 409806      1   0 17:46:06      -  0:00 /bin/sh /etc/init.crsd run
  oracle 454904 389122   0 17:45:41      -  0:00 /crs/app/oracle/product/crs_1/bin/ocssd.bin
    root 524338 418004   0 17:45:40      -  0:00 /crs/app/oracle/product/crs_1/bin/oprocd run -t 1000 -m 500
    root 569568 421918   0 19:02:19  pts/1  0:00 grep crs
[root@p520:/etc]#ps -ef|grep crs
  oracle 208974 323776   0 19:02:22      -  0:00 /crs/app/oracle/product/crs_1/bin/evmlogger.bin -o /crs/app/oracle/product/crs_1/evm/log/evmlogger.info -l /crs/app/oracle/product/crs_1/evm/log/evmlogger.log
    root 229546      1   1 17:45:37      -  0:01 /crs/app/oracle/product/crs_1/bin/crsd.bin reboot
  oracle 323776      1   0 17:45:53      -  0:00 /crs/app/oracle/product/crs_1/bin/evmd.bin
  oracle 389122 360648   0 17:45:41      -  0:00 /bin/sh -c ulimit -c unlimited; cd /crs/app/oracle/product/crs_1/log/p520/cssd;  /crs/app/oracle/product/crs_1/bin/ocssd  || exit $?
    root 409806      1   0 17:46:06      -  0:00 /bin/sh /etc/init.crsd run
  oracle 454904 389122   0 17:45:41      -  0:00 /crs/app/oracle/product/crs_1/bin/ocssd.bin
    root 524338 418004   0 17:45:40      -  0:00 /crs/app/oracle/product/crs_1/bin/oprocd run -t 1000 -m 500
    root 540718 421918   0 19:02:27  pts/1  0:00 grep crs

查看crs状态，都online,正常

[oracle@p520:/oracle/app/oracle]$crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora.p520.gsd   application    ONLINE    ONLINE    p520       
ora.p520.ons   application    ONLINE    ONLINE    p520       
ora.p520.vip   application    ONLINE    ONLINE    p520       
ora.p650.gsd   application    ONLINE    ONLINE    p650       
ora.p650.ons   application    ONLINE    ONLINE    p650       
ora.p650.vip   application    ONLINE    ONLINE    p650  

以下为metalink的详细解释

In this Document
  Symptoms
  Changes
  Cause
  Solution
  References

---

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1
This problem can occur on any platform.

Symptoms

2 node RAC, performing CRS 10.2.0.1 installation, failure at root.sh, CRS stack not started

WARNING: directory '/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: ndb1 ndb1-rac ndb1
node 2: ndb2 ndb2-rac ndb2
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
Now formatting voting device: /dev/rhdisk3
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Failure at final check of Oracle CRS stack.
10

Checking "ps -ef | grep init" during this 600 seconds period, see there is no crs related init processes, eg: init.crsd/init.evmd/init.cssd running. When no processes are detected, then this case apply.

Changes

New installation

Cause

This particular case is caused by the OS init system does not working.

" Failure at final check of Oracle CRS stack.
10"
means CRS daemon did not startup during 600 seconds period.

In root.sh script, it will add CRS related entry in /etc/inittab, run "init q" and expect those 3 CRS related daemon processes to start. With init system problem, none of these daemon processes spawned, this caused CRS process startup failure as they rely on init daemon process to start first.

This can be verified by adding a simple entry in /etc/inittab:
test:2:once:/usr/bin/echo "HELLO TEST" > /tmp/test.log
run "init q" as root user. If the init is working, then there should be a file /tmp/test.log generated.

Solution

Please consult with system administrator for init issue.

e.g. here the solution reference only valid for AIX platform.:
1. Starting the script. install_assist (AIX GUI utility Installation Assistance)
2. Updating for example the date, then exit install_assist properly
3. Reboot the system
After that daemon process in /etc/inittab started, CRS installation completed.

For other platforms, please consult your system admin or vendor for its solution.

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/7199859/viewspace-659985/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/7199859/viewspace-659985/