crs_stop -all 命令后遇到 UNKNOWN
crs_stop -all 命令后遇到 UNKNOWN
RAC中执行crs_stop -all后提示一些服务停不了,再用crs_stat -t查看状态的时候,停不掉的服务会显示状态是UNKNOWN,
服务在UNKNOWN状态下用 crs_start 不能启动服务
例如:
[oracle@rac1 ~]$ crs_stop -all
Attempting to stop `ora.rac1.gsd` on member `rac1`
Attempting to stop `ora.rac1.ons` on member `rac1`
Attempting to stop `ora.rac2.gsd` on member `rac2`
Attempting to stop `ora.ora9i.db` on member `rac1`
Attempting to stop `ora.rac2.ons` on member `rac2`
Stop of `ora.rac1.gsd` on member `rac1` succeeded.
Stop of `ora.rac2.gsd` on member `rac2` succeeded.
Stop of `ora.rac2.ons` on member `rac2` succeeded.
Stop of `ora.rac1.ons` on member `rac1` succeeded.
Stop of `ora.ora9i.db` on member `rac1` succeeded.
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.ora9i.ora9i1.inst` is already OFFLINE.
`ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.ora9i.ora9i2.inst` is already OFFLINE.
Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`
`ora.rac1.ASM1.asm` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
`ora.rac2.ASM2.asm` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`
`ora.rac1.ASM1.asm` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Attempting to stop `ora.rac1.vip` on member `rac1`
`ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`
`ora.rac2.ASM2.asm` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.vip` on member `rac2`
Stop of `ora.rac1.vip` on member `rac1` succeeded.
Stop of `ora.rac2.vip` on member `rac2` succeeded.
CRS-0216: Could not stop resource 'ora.ora9i.ora9i1.inst'.
CRS-0216: Could not stop resource 'ora.ora9i.ora9i2.inst'.
CRS-0216: Could not stop resource 'ora.rac1.ASM1.asm'.
CRS-0216: Could not stop resource 'ora.rac1.LISTENER_RAC1.lsnr'.
CRS-0216: Could not stop resource 'ora.rac1.vip'.
CRS-0216: Could not stop resource 'ora.rac2.ASM2.asm'.
CRS-0216: Could not stop resource 'ora.rac2.LISTENER_RAC2.lsnr'.
CRS-0216: Could not stop resource 'ora.rac2.vip'.
在UNKNOWN状态下,使用crs_start -all命令不会把RAC启动起来,启动的时候会报错。
[oracle@rac1 ~]$ crs_start -all
Attempting to start `ora.rac1.vip` on member `rac1`
Attempting to start `ora.rac2.vip` on member `rac2`
Attempting to start `ora.rac2.gsd` on member `rac2`
Attempting to start `ora.rac2.ons` on member `rac2`
Attempting to start `ora.rac1.gsd` on member `rac1`
Attempting to start `ora.rac1.ons` on member `rac1`
`ora.rac2.gsd` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.rac2.ons` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to start `ora.ora9i.db` on member `rac1`
`ora.rac1.ons` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.rac1.gsd` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.ora9i.db` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Start of `ora.rac1.vip` on member `rac1` succeeded.
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.rac1.ASM1.asm'
Start of `ora.rac2.vip` on member `rac2` succeeded.
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.rac2.ASM2.asm'
CRS-0215: Could not start resource 'ora.ora9i.db'.
CRS-0223: Resource 'ora.ora9i.ora9i1.inst' has placement error.
CRS-0223: Resource 'ora.ora9i.ora9i2.inst' has placement error.
CRS-0215: Could not start resource 'ora.rac1.gsd'.
CRS-0215: Could not start resource 'ora.rac1.ons'.
CRS-0215: Could not start resource 'ora.rac2.gsd'.
CRS-0215: Could not start resource 'ora.rac2.ons'.
此时crs_stat -t的状态还是有很多的UNKNOWN
(烦、烦、烦,rac中真奇怪,正常启动关闭服务时,
状态只有online和offline两种(启动与未启动),结果RAC中给你来了个UNKNOWN):
[oracle@rac1 ~]$ crs_stat -t
Name
------------------------------------------------------------
ora.ora9i.db
ora....i1.inst application
ora....i2.inst application
ora....SM1.asm application
ora....C1.lsnr application
ora.rac1.gsd
ora.rac1.ons
ora.rac1.vip
ora....SM2.asm application
ora....C2.lsnr application
ora.rac2.gsd
ora.rac2.ons
ora.rac2.vip
[oracle@rac1 ~]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Thu Aug 7 13:42:36 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> conn /as sysdba
Connected to an idle instance.
显示数据库已经连接不上;
此时可有两种方法来解决:
方法一:先禁用crs服务,后启用crs服务
(相当于从新启动系统,因为crs是随系统自动启动的),需要在两台节点上都执行;
具体步骤:
节点1:
1、[root@rac1 ~]# /etc/init.d/init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
2、[root@rac1 ~]# su - oracle
[oracle@rac1 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
3、[oracle@rac1 ~]$ exit
logout
[root@rac1 ~]# /etc/init.d/init.crs start
Startup will be queued to init within 90 seconds.
4、[root@rac1 ~]# su - oracle
[oracle@rac1 ~]$ crs_stat -t
Name
------------------------------------------------------------
ora.ora9i.db
ora....i1.inst application
ora....i2.inst application
ora....SM1.asm application
ora....C1.lsnr application
ora.rac1.gsd
ora.rac1.ons
ora.rac1.vip
ora....SM2.asm application
ora....C2.lsnr application
ora.rac2.gsd
ora.rac2.ons
ora.rac2.vip
[oracle@rac1 ~]$
节点2:
1、[root@rac2 ~]# /etc/init.d/init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
2、[root@rac2 ~]# su - oracle
[oracle@rac2 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
[oracle@rac2 ~]$ exit
logout
3、[root@rac2 ~]# /etc/init.d/init.crs start
Startup will be queued to init within 90 seconds.
4、[root@rac2 ~]# su - oracle
[oracle@rac2 ~]$ crs_stat -t
Name
------------------------------------------------------------
ora.ora9i.db
ora....i1.inst application
ora....i2.inst application
ora....SM1.asm application
ora....C1.lsnr application
ora.rac1.gsd
ora.rac1.ons
ora.rac1.vip
ora....SM2.asm application
ora....C2.lsnr application
ora.rac2.gsd
ora.rac2.ons
ora.rac2.vip
[oracle@rac2 ~]$
方法二:可以使用crs_stop -f先强制停止服务,在启动服务(在一台节点上运行即可);
具体步骤:
[oracle@rac2 ~]$ crs_stop -f ora.rac1.ons
Attempting to stop `ora.rac1.ons` on member `rac1`
Stop of `ora.rac1.ons` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.ons
Attempting to start `ora.rac1.ons` on member `rac1`
Start of `ora.rac1.ons` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.ons
Attempting to stop `ora.rac2.ons` on member `rac2`
Stop of `ora.rac2.ons` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.ons
Attempting to start `ora.rac2.ons` on member `rac2`
Start of `ora.rac2.ons` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac1.gsd
Attempting to stop `ora.rac1.gsd` on member `rac1`
Stop of `ora.rac1.gsd` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.gsd
Attempting to start `ora.rac1.gsd` on member `rac1`
Start of `ora.rac1.gsd` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.gsd
Attempting to stop `ora.rac2.gsd` on member `rac2`
Stop of `ora.rac2.gsd` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.gsd
Attempting to start `ora.rac2.gsd` on member `rac2`
Start of `ora.rac2.gsd` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac1.LISTENER_RAC1.lsnr
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.LISTENER_RAC1.lsnr
Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.LISTENER_RAC2.lsnr
Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Stop of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.LISTENER_RAC2.lsnr
Attempting to start `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Start of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac1.ASM1.asm
Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`
Stop of `ora.rac1.ASM1.asm` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.ASM1.asm
Attempting to start `ora.rac1.ASM1.asm` on member `rac1`
Start of `ora.rac1.ASM1.asm` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.ASM2.asm
Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`
Stop of `ora.rac2.ASM2.asm` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.ASM2.asm
Attempting to start `ora.rac2.ASM2.asm` on member `rac2`
Start of `ora.rac2.ASM2.asm` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.ora9i.ora9i1.inst
`ora.ora9i.ora9i1.inst` is already OFFLINE.
CRS-0216: Could not stop resource 'ora.ora9i.ora9i1.inst'.
[oracle@rac2 ~]$ crs_start -f ora.ora9i.ora9i1.inst
Attempting to start `ora.ora9i.ora9i1.inst` on member `rac1`
Start of `ora.ora9i.ora9i1.inst` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.ora9i.ora9i2.inst
`ora.ora9i.ora9i2.inst` is already OFFLINE.
CRS-0216: Could not stop resource 'ora.ora9i.ora9i2.inst'.
[oracle@rac2 ~]$ crs_start -f ora.ora9i.ora9i2.inst
Attempting to start `ora.ora9i.ora9i2.inst` on member `rac2`
Start of `ora.ora9i.ora9i2.inst` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.ora9i.db
Attempting to stop `ora.ora9i.db` on member `rac2`
Stop of `ora.ora9i.db` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.ora9i.db
Attempting to start `ora.ora9i.db` on member `rac1`
Start of `ora.ora9i.db` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stat -t
Name
------------------------------------------------------------
ora.ora9i.db
ora....i1.inst application
ora....i2.inst application
ora....SM1.asm application
ora....C1.lsnr application
ora.rac1.gsd
ora.rac1.ons
ora.rac1.vip
ora....SM2.asm application
ora....C2.lsnr application
ora.rac2.gsd
ora.rac2.ons
ora.rac2.vip
总结:在rac环境中如果使用crs_stop -all去停rac服务时,很有可能会出现crs中的一些服务的状态是UNKNOWN,这时我们可以采用两种方法去重新启动crs服务:
1:转换到root用户下用/etc/init.d/init.crs stop先禁用crs,然后再用/etc/init.d/init.crs start去启用crs,启用crs后会自动启动crs的一系列服务,注意此种方法需要在两台节点上都执行;
2:是用crs_stop -f 参数把crs中状态是UNKNOWN的服务关掉,然后再用crs_start -f (加一个-f的参数)启动所有的服务就可以。参考:http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=31
以下要分别在两个节点上执行:
在启动的过程中最好检测着crs、ASM和数据库的日志:
crs日志:
[oracle@rac1 ~]$ tail -f /u01/app/oracle/product/10.2.0/crs_1/log/rac1/alertrac1.log
[oracle@rac2 ~]$ tail -f /u01/app/oracle/product/10.2.0/crs_1/log/rac2/alertrac2.log
[oracle@rac1 ~]$ tail -f /u01/app/oracle/admin/+ASM/bdump/alert_+ASM1.log
[oracle@rac2 ~]$ tail -f /u01/app/oracle/admin/+ASM/bdump/alert_+ASM2.log
数据库日志:
[oracle@rac1 ~]$ tail -f /u01/app/oracle/admin/ora9i/bdump/alert_ora9i1.log
[oracle@rac2 ~]$ tail -f /u01/app/oracle/admin/ora9i/bdump/alert_ora9i2.log