oracle crs 4639,CRS-4639: Could not contact Oracle High Availability Services解决方案集

在启动asm实例的时候报如下错误：

[grid@b1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Thu Sep 12 18:14:13 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup

ORA-01078: failure in processing system parameters

ORA-29701: unable to connect to Cluster Synchronization Service

然后用crsctl check css检查的时候报如下错误：

[grid@b1 ~]$ crsctl check css

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Check failed, or completed with errors.

解决CRS-4639: Could not contact Oracle High Availability Services过程如下：

[root@b1 grid]# cd /u01/app/11.2.0/grid/crs/install

[root@b1 install]#  ./roothas.pl -deconfig -force -verbose

2013-09-12 19:25:05: Checking for super user privileges

2013-09-12 19:25:05: User has super user privileges

2013-09-12 19:25:05: Parsing the host name

Using configuration parameter file: ./crsconfig_params

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-4639: Could not contact Oracle High Availability Services

CRS-4000: Command Delete failed, or completed with errors.

Failure at scls_scr_getval with code 1

Internal Error Information:

Category: -2

Operation: opendir

Location: scrsearch1

Other: cant open scr home dir scls_scr_getval

System Dependent Information: 2

CRS-4544: Unable to connect to OHAS

CRS-4000: Command Stop failed, or completed with errors.

ACFS-9200: Supported

Successfully deconfigured Oracle Restart stack

[root@b1 install]# cd /u01/app/11.2.0/grid/

[root@b1 grid]# ./root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]: y

Copying dbhome to /usr/local/bin ...

The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]: y

Copying oraenv to /usr/local/bin ...

The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]: y

Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

2013-09-12 19:27:31: Checking for super user privileges

2013-09-12 19:27:31: User has super user privileges

2013-09-12 19:27:31: Parsing the host name

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

LOCAL ADD MODE

Creating OCR keys for user 'grid', privgrp 'oinstall'..

Operation successful.

CRS-4664: Node b1 successfully pinned.

Adding daemon to inittab

CRS-4123: Oracle High Availability Services has been started.

ohasd is starting

b1     2013/09/12 19:29:12     /u01/app/11.2.0/grid/cdata/b1/backup_20130912_192912.olr

Successfully configured Oracle Grid Infrastructure for a Standalone Server

Updating inventory properties for clusterware

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

'UpdateNodeList' was successful.

[grid@b1 ~]$ crs_stat -t

Name           Type           Target    State     Host

------------------------------------------------------------

ora.cssd       ora.cssd.type  OFFLINE   OFFLINE

ora.diskmon    ora....on.type OFFLINE   OFFLINE

[grid@b1 ~]$ crs_start -all

Attempting to start `ora.diskmon` on member `b1`

Attempting to start `ora.cssd` on member `b1`

Start of `ora.diskmon` on member `b1` succeeded.

Start of `ora.cssd` on member `b1` succeeded.

[grid@b1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Thu Sep 12 19:34:50 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup

ASM instance started

Total System Global Area  283930624 bytes

Fixed Size                  2212656 bytes

Variable Size             256552144 bytes

ASM Cache                  25165824 bytes

ASM diskgroups mounted

ASM diskgroups volume enabled

-----------&gt>第二种解决方法:

[grid@vm11gr2] /home/grid> sqlplus "/as sysasm"

SQL*Plus:

Release 11.2.0.1.0 Production on Sun Oct 25 10:16:21 2009

Copyright

(c) 1982, 2009, Oracle. All rights reserved.

Connected to

an idle instance.

SQL> startup

ORA-01078:

failure in processing system parameters

ORA-29701: unable

to connect to Cluster Synchronization Service

SQL>

无法连接到CSS服务上.到操作系统上检查一下看看

[grid@vm11gr2] /home/grid> crsctl check css

CRS-4530:

Communications failure contacting

Cluster Synchronization Services daemon

[grid@vm11gr2] /home/grid>

[grid@vm11gr2] /home/grid> ps -ef|grep cssd

果然没有CSS的服务daemon进程,再看一下HAS(High Availability Service)的状态

[grid@vm11gr2] /home/grid> crsctl check has

CRS-4638:

Oracle High Availability Services is online

[grid@vm11gr2] /home/grid> ps -ef|grep d.bin

grid 5886 1

0 10:06 ? 00:00:01 /u01/app/grid/product/11.2/grid/bin/ohasd.bin

reboot

[grid@vm11gr2] /home/grid>

发现HAS的服务确实启动了的,而ora.cssd和ora.diskmon这2个服务是依赖于HAS维护的.

进一步查看各资源的状态

[grid@vm11gr2] /home/grid> crs_stat -t

Name

Type

Target    State

Host

--------------------------------------------------------------

ora.FLASH_DATA.dg

ora.diskgroup.type   OFFLINE

OFFLINE vm11gr2

ora.SYS_DATA.dg

ora.diskgroup.type   OFFLINE

OFFLINE vm11gr2

ora.asm

ora.asm.type

OFFLINE OFFLINE vm11gr2

ora.cssd

ora.cssd.type

OFFLINE OFFLINE vm11gr2

ora.diskmon

ora.diskmon.type

OFFLINE OFFLINE vm11gr2

[grid@vm11gr2] /home/grid>

[grid@vm11gr2] /home/grid> crsctl status resource -t

--------------------------------------------------------------------------------

NAME

TARGET

STATE

SERVER

STATE_

DETAILS

--------------------------------------------------------------------------------

Local

Resources

--------------------------------------------------------------------------------

ora.FLASH_DATA.dg

OFFLINE

OFFLINE

vm11gr2

ora.SYS_DATA.dg

OFFLINE

OFFLINE

vm11gr2

ora.asm

OFFLINE

OFFLINE

vm11gr2

--------------------------------------------------------------------------------

Cluster

Resources

--------------------------------------------------------------------------------

ora.cssd

1         OFFLINE

OFFLINE

ora.diskmon

1         OFFLINE

OFFLINE

再看一下ora.cssd和ora.diskmon的属性

[grid@vm11gr2] /home/grid> crs_stat -p

ora.cssd

NAME=ora.cssd

TYPE=ora.cssd.type

ACTION_SCRIPT=

ACTIVE_PLACEMENT=0

AUTO_START=never

CHECK_INTERVAL=30

DESCRIPTION="Resource type for CSSD"

FAILOVER_DELAY=0

FAILURE_INTERVAL=3

FAILURE_THRESHOLD=5

HOSTING_MEMBERS=

PLACEMENT=balanced

RESTART_ATTEMPTS=5

SCRIPT_TIMEOUT=600

START_TIMEOUT=600

STOP_TIMEOUT=900

UPTIME_THRESHOLD=1m

[grid@vm11gr2] /home/grid> crs_stat -p ora.diskmon

NAME=ora.diskmon

TYPE=ora.diskmon.type

ACTION_SCRIPT=

ACTIVE_PLACEMENT=0

AUTO_START=never

CHECK_INTERVAL=20

DESCRIPTION="Resource type for Diskmon"

FAILOVER_DELAY=0

FAILURE_INTERVAL=3

FAILURE_THRESHOLD=5

HOSTING_MEMBERS=

PLACEMENT=balanced

RESTART_ATTEMPTS=10

SCRIPT_TIMEOUT=60

START_TIMEOUT=60

STOP_TIMEOUT=60

UPTIME_THRESHOLD=5s

[grid@vm11gr2] /home/grid>

到这里基本就找到了原因了,可以看到这两个资源的AUTO_START属性默认都设置为never,也就是说他们不会随着HAS服务的启动而自动启动的,

尽管默认情况下HAS服务是开机自动启动的.好了，那我们就手动启动一下吧:

[grid@vm11gr2] /home/grid> crsctl start resource

ora.cssd

CRS-2672:

Attempting to start 'ora.cssd' on 'vm11gr2'

CRS-2679:

Attempting to clean 'ora.diskmon' on 'vm11gr2'

CRS-2681:

Clean of 'ora.diskmon' on 'vm11gr2' succeeded

CRS-2672:

Attempting to start 'ora.diskmon' on 'vm11gr2'

CRS-2676:

Start of 'ora.diskmon' on 'vm11gr2' succeeded

CRS-2676:

Start of 'ora.cssd' on 'vm11gr2' succeeded

[grid@vm11gr2] /home/grid>

注:ora.cssd和ora.diskmon这两个服务是有依赖关系的，启动哪个都会把两个都起来.

[grid@vm11gr2] /home/grid> crs_stat -t

Name

Type

Target    State

Host

--------------------------------------------------------------

ora.FLASH_DATA.dg

ora.diskgroup.type   OFFLINE

OFFLINE vm11gr2

ora.SYS_DATA.dg

ora.diskgroup.type   OFFLINE

OFFLINE vm11gr2

ora.asm

ora.asm.type

OFFLINE OFFLINE vm11gr2

ora.cssd

ora.cssd.type

ONLINE  ONLINE  vm11gr2

ora.diskmon

ora.diskmon.type

ONLINE  ONLINE  vm11gr2

[grid@vm11gr2] /home/grid>

CSS服务起来了，重启动asm

instance

[grid@vm11gr2] /home/grid> sqlplus "/as sysasm"

SQL*Plus:

Release 11.2.0.1.0 Production on Sun Oct 25 10:30:03 2009

Copyright

(c) 1982, 2009, Oracle. All rights reserved.

Connected to

an idle instance.

SQL> startup

ASM instance

started

Total System

Global Area 284565504 bytes

Fixed Size

1336036 bytes

Variable

Size 258063644 bytes

ASM Cache

25165824 bytes

ASM

diskgroups mounted

SQL> exit

Disconnected

from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -

Production

With the

Automatic Storage Management option

[grid@vm11gr2] /home/grid> crs_stat -t

Name

Type

Target    State

Host

--------------------------------------------------------------

ora.FLASH_DATA.dg

ora.diskgroup.type

ONLINE  ONLINE  vm11gr2

ora.SYS_DATA.dg

ora.diskgroup.type

ONLINE  ONLINE  vm11gr2

ora.asm

ora.asm.type

ONLINE  ONLINE  vm11gr2

ora.cssd

ora.cssd.type

ONLINE  ONLINE  vm11gr2

ora.diskmon

ora.diskmon.type

ONLINE  ONLINE  vm11gr2

[grid@vm11gr2] /home/grid>

tips

1)默认情况下HAS(High Availability Service)是自动启动的.通过如下命令可以取消和启用自动启动

crsctl disable has

crsctl enable has

2)HAS手动启动和停止

crsctl start has

crsctl stop has

3)查看HAS的状态

crsctl check has

4)如果想让ora.css和ora.diskmon服务随着HAS的启动而自动启动，那么你可以这两个服务的AUTO_START属性

crsctl modify resource "ora.cssd" -attr "AUTO_START=1"

or

crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"

5)如果想取消ora.css和ora.diskmon的Auto start

crsctl modify resource "ora.cssd" -attr "AUTO_START=never"

crsctl modify resource "ora.diskmon" -attr

"AUTO_START=never"

------&gt>11gR2启动ASM实例时遭遇ORA-29701

[grid@vm11gr2] /home/grid>sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.1.0 Production on Sun Oct 25 10:16:21 2009

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service SQL>

无法连接到CSS服务上.到操作系统上检查一下看看

[grid@vm11gr2] /home/grid> crsctl check css CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

[grid@vm11gr2] /home/grid>

[grid@vm11gr2] /home/grid> ps -ef|grep cssd

果然没有CSS的服务daemon进程,再看一下HAS(High Availability Service)的状态

[grid@vm11gr2] /home/grid> crsctl check has CRS-4638: Oracle High Availability Services isonline [grid@vm11gr2] /home/grid> ps -ef|grep d.bin grid 5886 1 0 10:06 ? 00:00:01 /u01/app/grid/product/11.2/grid/bin/ohasd.bin reboot

[grid@vm11gr2] /home/grid>

发现HAS的服务确实启动了的,而ora.cssd和ora.diskmon这2个服务是依赖于HAS维护的. 进一步查看各资源的状态 [grid@vm11gr2] /home/grid> crs_stat -t Name Type Target State Host

--------------------------------------------------------------

ora.FLASH_DATA.dg ora.diskgroup.type OFFLINE OFFLINE vm11gr2

ora.SYS_DATA.dg ora.diskgroup.type OFFLINE OFFLINE vm11gr2

ora.asm ora.asm.type OFFLINE OFFLINE vm11gr2

ora.cssd ora.cssd.type OFFLINE OFFLINE vm11gr2

ora.diskmon ora.diskmon.type OFFLINE OFFLINE vm11gr2

[grid@vm11gr2] /home/grid>

[grid@vm11gr2] /home/grid> crsctl status resource -t --------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_ DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.FLASH_DATA.dg

OFFLINE OFFLINE vm11gr2

ora.SYS_DATA.dg

OFFLINE OFFLINE vm11gr2

ora.asm

OFFLINE OFFLINE vm11gr2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

1 OFFLINE OFFLINE ora.diskmon

1 OFFLINE OFFLINE 再看一下ora.cssd和ora.diskmon的属性 [grid@vm11gr2] /home/grid> crs_stat -p ora.cssd

NAME=ora.cssd

TYPE=ora.cssd.type

ACTION_SCRIPT=

ACTIVE_PLACEMENT=0 AUTO_START=never CHECK_INTERVAL=30

DESCRIPTION="Resource type for CSSD"

FAILOVER_DELAY=0

FAILURE_INTERVAL=3

FAILURE_THRESHOLD=5

HOSTING_MEMBERS=

PLACEMENT=balanced

RESTART_ATTEMPTS=5

SCRIPT_TIMEOUT=600

START_TIMEOUT=600

STOP_TIMEOUT=900

UPTIME_THRESHOLD=1m

[grid@vm11gr2] /home/grid> crs_stat -p ora.diskmon NAME=ora.diskmon

TYPE=ora.diskmon.type

ACTION_SCRIPT=

ACTIVE_PLACEMENT=0 AUTO_START=never CHECK_INTERVAL=20

DESCRIPTION="Resource type for Diskmon"

FAILOVER_DELAY=0

FAILURE_INTERVAL=3

FAILURE_THRESHOLD=5

HOSTING_MEMBERS=

PLACEMENT=balanced

RESTART_ATTEMPTS=10

SCRIPT_TIMEOUT=60

START_TIMEOUT=60

STOP_TIMEOUT=60

UPTIME_THRESHOLD=5s

[grid@vm11gr2] /home/grid>

到这里基本就找到了原因了,可以看到这两个资源的AUTO_START属性默认都设置为never,也就是说他们不会随着HAS服务的启动而自动启动的, 尽管默认情况下HAS服务是开机自动启动的.好了，那我们就手动启动一下吧: [grid@vm11gr2] /home/grid> crsctl start resource ora.cssd CRS-2672: Attempting to start 'ora.cssd' on 'vm11gr2'

CRS-2679: Attempting to clean 'ora.diskmon' on 'vm11gr2'

CRS-2681: Clean of 'ora.diskmon' on 'vm11gr2' succeeded

CRS-2672: Attempting to start 'ora.diskmon' on 'vm11gr2'

CRS-2676: Start of 'ora.diskmon' on 'vm11gr2' succeeded

CRS-2676: Start of 'ora.cssd' on 'vm11gr2' succeeded

[grid@vm11gr2] /home/grid> 注:ora.cssd和ora.diskmon这两个服务是有依赖关系的，启动哪个都会把两个都起来.

[grid@vm11gr2] /home/grid> crs_stat -t Name Type Target State Host

--------------------------------------------------------------

ora.FLASH_DATA.dg ora.diskgroup.type OFFLINE OFFLINE vm11gr2

ora.SYS_DATA.dg ora.diskgroup.type OFFLINE OFFLINE vm11gr2

ora.asm ora.asm.type OFFLINE OFFLINE vm11gr2

ora.cssd ora.cssd.type ONLINE ONLINE vm11gr2

ora.diskmon ora.diskmon.type ONLINE ONLINE vm11gr2

[grid@vm11gr2] /home/grid> CSS服务起来了，重启动asm instance [grid@vm11gr2] /home/grid> sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.1.0 Production on Sun Oct 25 10:30:03 2009

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup ASM instance started

Total System Global Area 284565504 bytes

Fixed Size 1336036 bytes

Variable Size 258063644 bytes

ASM Cache 25165824 bytes

ASM diskgroups mounted

SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production

With the Automatic Storage Management option

[grid@vm11gr2] /home/grid>crs_stat -t Name Type Target State Host

--------------------------------------------------------------

ora.FLASH_DATA.dg ora.diskgroup.type ONLINE ONLINE vm11gr2

ora.SYS_DATA.dg ora.diskgroup.type ONLINE ONLINE vm11gr2

ora.asm ora.asm.type ONLINE ONLINE vm11gr2

ora.cssd ora.cssd.type ONLINE ONLINE vm11gr2

ora.diskmon ora.diskmon.type ONLINE ONLINE vm11gr2

[grid@vm11gr2] /home/grid> tips 1)默认情况下HAS(High Availability Service)是自动启动的.通过如下命令可以取消和启用自动启动

crsctl disable has

crsctl enable has

2)HAS手动启动和停止

crsctl start has

crsctl stop has

3)查看HAS的状态

crsctl check has

4)如果想让ora.css和ora.diskmon服务随着HAS的启动而自动启动，那么你可以这两个服务的AUTO_START属性

crsctl modify resource "ora.cssd" -attr "AUTO_START=1"

or

crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"

5)如果想取消ora.css和ora.diskmon的Auto start

crsctl modify resource "ora.cssd" -attr "AUTO_START=never"

crsctl modify resource "ora.diskmon" -attr "AUTO_START=never" ==================================================================================

自己遇到的情况是：

[grid@vm11gr2] /home/grid>sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.1.0 Production on Sun Oct 25 10:16:21 2009

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+RAC_DISK/RACDB/spfileRACDB.ora'

ORA-17503: ksfdopn:2 Failed to open file +RAC_DISK/RACDB/spfileRACDB.ora

ORA-15077: could not locate ASM instance serving a required diskgroup

ORA-29701: unable to connect to Cluster Synchronization Service

----&gt>感同身受篇:rac02数据库异常停机报错:

Error: received an ABORT event from the Cluster Synchronization Service

Errors in file /u01/app/oracle/diag/rdbms/sun/sun/trace/sun_lmon_25888.trc:

ORA-29701: unable to connect to Cluster Synchronization Service

LMON (ospid: 25888): terminating the instance due to error 29701

Wed Nov 19 16:17:51 2014

System state dump is made for local instance

System State dumped to trace file /u01/app/oracle/diag/rdbms/sun/sun/trace/sun_diag_25864.trc

Instance terminated by LMON, pid = 25888

Wed Nov 19 23:38:23 2014

Starting ORACLE instance (normal)

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Interface type 1 eth1 172.16.2.0 configured from GPnP Profile for use as a cluster interconnect

Interface type 1 eth0 192.168.2.0 configured from GPnP Profile for use as a public interface

Picked latch-free SCN scheme 3

Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST

Autotune of undo retention is turned on.

IMODE=BR

ILAT =88

LICENSE_MAX_USERS = 0

SYS auditing is disabled

NOTE: Volume support  enabled

Starting up:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP, Data Mining

and Real Application Testing options.

--sun_lmon_25888.trc内容:

[oracle@rac02 trace]$ cat sun_lmon_25888.trc

Trace file /u01/app/oracle/diag/rdbms/bus/bus/trace/sun_lmon_25888.trc

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1

System name:    Linux

Node name:      rac02

Release:        2.6.18-308.el5

Version:        #1 SMP Fri Jan 27 17:17:51 EST 2012

Machine:        x86_64

Instance name: sun

Redo thread mounted by this instance: 1

Oracle process number: 11

Unix process pid: 25888, image: oracle@rac02 (LMON)

*** 2014-11-19 16:17:51.745

*** SESSION ID:(276.1) 2014-11-19 16:17:51.745

*** CLIENT ID:() 2014-11-19 16:17:51.745

*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-19 16:17:51.745

*** MODULE NAME:() 2014-11-19 16:17:51.745

*** ACTION NAME:() 2014-11-19 16:17:51.745

*** TRACE FILE RECREATED AFTER BEING REMOVED ***

2014-11-19 16:17:51.744: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1

2014-11-19 16:17:51.745: [ CSSCLNT]clssgsGroupGetStatus:  communications failed (0/3/-1)

2014-11-19 16:17:51.745: [ CSSCLNT]clssgsGroupGetStatus: returning 8

kgxgnpstat: received ABORT event from CLSS

kjxggpoll: received an abort event from DBALL_DB

Error: Cluster Synchronization Service aborts the instance (error 6)

LMON caught an error 29701 in the main loop

error 29701 detected in background process

ORA-29701: unable to connect to Cluster Synchronization Service

*** 2014-11-19 16:17:51.754

LMON (ospid: 25888): terminating the instance due to error 29701

ksuitm: waiting up to [5] seconds before killing DIAG(25864)