ORA-29701: unable to connect to Cluster Synchronization Service

[grid@test170 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Sep 3 14:36:19 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
SQL> exit
Disconnected
[grid@test170 ~]$ crsctl check css
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
[grid@test170 ~]$ ps -ef|grep cssd
grid      3530  3372  0 14:37 pts/1    00:00:00 grep cssd
[grid@test170 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@test170 ~]$  ps -ef|grep d.bin
grid      3032     1  0 13:33 ?        00:00:05 /g01/app/oracle/product/grid/bin/ohasd.bin reboot
grid      3237     1  0 13:34 ?        00:00:02 /g01/app/oracle/product/grid/bin/oraagent.bin
grid      3253     1  0 13:34 ?        00:00:00 /g01/app/oracle/product/grid/bin/evmd.bin
grid      3282  3253  0 13:34 ?        00:00:00 /g01/app/oracle/product/grid/bin/evmlogger.bin -o /g01/app/oracle/product/grid/evm/log/evmlogger.info -l /g01/app/oracle/product/grid/evm/log/evmlogger.log
grid      3537  3372  0 14:37 pts/1    00:00:00 grep d.bin
[grid@test170 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type OFFLINE   OFFLINE               
ora.asm        ora.asm.type   OFFLINE   OFFLINE               
ora.cssd       ora.cssd.type  ONLINE    OFFLINE               
ora.diskmon    ora....on.type OFFLINE   OFFLINE               
ora.evmd       ora.evm.type   ONLINE    ONLINE    test170     
ora.ons        ora.ons.type   OFFLINE   OFFLINE               
ora.tcyang.db  ora....se.type OFFLINE   OFFLINE               
[grid@test170 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               OFFLINE OFFLINE      test170                                      
ora.asm
               OFFLINE OFFLINE      test170                  Instance Shutdown   
ora.ons
               OFFLINE OFFLINE      test170                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  OFFLINE                                                   
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  ONLINE       test170                                      
ora.tcyang.db
      1        OFFLINE OFFLINE                               Instance Shutdown   
[grid@test170 ~]$ crs_stat -p ora.cssd
NAME=ora.cssd
TYPE=ora.cssd.type
ACTION_SCRIPT=
ACTIVE_PLACEMENT=0
AUTO_START=never
CHECK_INTERVAL=30
DESCRIPTION="Resource type for CSSD"
FAILOVER_DELAY=0
FAILURE_INTERVAL=3
FAILURE_THRESHOLD=5
HOSTING_MEMBERS=
PLACEMENT=balanced
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=600
STOP_TIMEOUT=900
UPTIME_THRESHOLD=1m

[grid@test170 ~]$ crs_stat -p ora.diskmon
NAME=ora.diskmon
TYPE=ora.diskmon.type
ACTION_SCRIPT=
ACTIVE_PLACEMENT=0
AUTO_START=never
CHECK_INTERVAL=3
DESCRIPTION="Resource type for Diskmon"
FAILOVER_DELAY=0
FAILURE_INTERVAL=3
FAILURE_THRESHOLD=5
HOSTING_MEMBERS=
PLACEMENT=balanced
RESTART_ATTEMPTS=10
SCRIPT_TIMEOUT=60
START_TIMEOUT=600
STOP_TIMEOUT=60
UPTIME_THRESHOLD=5s

[grid@test170 ~]$ crsctl start resource ora.cssd
CRS-2672: Attempting to start 'ora.cssd' on 'test170'
CRS-2672: Attempting to start 'ora.diskmon' on 'test170'
CRS-2676: Start of 'ora.diskmon' on 'test170' succeeded
CRS-2676: Start of 'ora.cssd' on 'test170' succeeded
[grid@test170 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type OFFLINE   OFFLINE               
ora.asm        ora.asm.type   OFFLINE   OFFLINE               
ora.cssd       ora.cssd.type  ONLINE    ONLINE    test170     
ora.diskmon    ora....on.type OFFLINE   OFFLINE               
ora.evmd       ora.evm.type   ONLINE    ONLINE    test170     
ora.ons        ora.ons.type   OFFLINE   OFFLINE               
ora.tcyang.db  ora....se.type OFFLINE   OFFLINE               
[grid@test170 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@test170 ~]$ crsctl --help
Usage: crsctl add       - add a resource, type or other entity
       crsctl check     - check a service, resource or other entity
       crsctl config    - output autostart configuration
       crsctl debug     - obtain or modify debug state
       crsctl delete    - delete a resource, type or other entity
       crsctl disable   - disable autostart
       crsctl discover  - discover DHCP server
       crsctl enable    - enable autostart
       crsctl get       - get an entity value
       crsctl getperm   - get entity permissions
       crsctl lsmodules - list debug modules
       crsctl modify    - modify a resource, type or other entity
       crsctl query     - query service state
       crsctl pin       - pin the nodes in the node list
       crsctl relocate  - relocate a resource, server or other entity
       crsctl replace   - replaces the location of voting files
       crsctl release   - release a DHCP lease
       crsctl request   - request a DHCP lease
       crsctl setperm   - set entity permissions
       crsctl set       - set an entity value
       crsctl start     - start a resource, server or other entity
       crsctl status    - get status of a resource or other entity
       crsctl stop      - stop a resource, server or other entity
       crsctl unpin     - unpin the nodes in the node list
       crsctl unset     - unset an entity value, restoring its default
[grid@test170 ~]$ crsctl start resource ora.asm
CRS-2672: Attempting to start 'ora.asm' on 'test170'
CRS-2676: Start of 'ora.asm' on 'test170' succeeded
[grid@test170 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    test170     
ora.asm        ora.asm.type   ONLINE    ONLINE    test170     
ora.cssd       ora.cssd.type  ONLINE    ONLINE    test170     
ora.diskmon    ora....on.type OFFLINE   OFFLINE               
ora.evmd       ora.evm.type   ONLINE    ONLINE    test170     
ora.ons        ora.ons.type   OFFLINE   OFFLINE               
ora.tcyang.db  ora....se.type OFFLINE   OFFLINE               
[grid@test170 ~]$ crsctl start resource ora.ons
CRS-2501: Resource 'ora.ons' is disabled
CRS-4000: Command Start failed, or completed with errors.
[grid@test170 ~]$ crsctl start resource ora.diskmon
CRS-2672: Attempting to start 'ora.diskmon' on 'test170'
CRS-2676: Start of 'ora.diskmon' on 'test170' succeeded
[grid@test170 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    test170     
ora.asm        ora.asm.type   ONLINE    ONLINE    test170     
ora.cssd       ora.cssd.type  ONLINE    ONLINE    test170     
ora.diskmon    ora....on.type OFFLINE   OFFLINE               
ora.evmd       ora.evm.type   ONLINE    ONLINE    test170     
ora.ons        ora.ons.type   OFFLINE   OFFLINE               
ora.tcyang.db  ora....se.type OFFLINE   OFFLINE               
[grid@test170 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Sep 3 14:45:02 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option

SQL> startup
ORA-01031: insufficient privileges
SQL> exit        
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option
[grid@test170 ~]$ asmcmd
ASMCMD> startup
ORA-01081: cannot start already-running ORACLE - shut it down first
ASMCMD> shutdown immediate;
ASM diskgroups dismounted
ASM instance shutdown
Connected to an idle instance.
ASMCMD> !csr_stat -t
sh: csr_stat: command not found
ASMCMD> exit
[grid@test170 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type OFFLINE   OFFLINE               
ora.asm        ora.asm.type   OFFLINE   OFFLINE               
ora.cssd       ora.cssd.type  ONLINE    ONLINE    test170     
ora.diskmon    ora....on.type OFFLINE   OFFLINE               
ora.evmd       ora.evm.type   ONLINE    ONLINE    test170     
ora.ons        ora.ons.type   OFFLINE   OFFLINE               
ora.tcyang.db  ora....se.type OFFLINE   OFFLINE               
[grid@test170 ~]$ asmcmd
Connected to an idle instance.
ASMCMD> startup
ASM instance started

Total System Global Area  283930624 bytes
Fixed Size                  2227664 bytes
Variable Size             256537136 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
ASMCMD> exit
[grid@test170 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    test170     
ora.asm        ora.asm.type   ONLINE    ONLINE    test170     
ora.cssd       ora.cssd.type  ONLINE    ONLINE    test170     
ora.diskmon    ora....on.type OFFLINE   OFFLINE               
ora.evmd       ora.evm.type   ONLINE    ONLINE    test170     
ora.ons        ora.ons.type   OFFLINE   OFFLINE               
ora.tcyang.db  ora....se.type OFFLINE   OFFLINE               
[grid@test170 ~]$ su - oracle
Password: 
[oracle@test170 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Sep 3 14:46:30 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area  626327552 bytes
Fixed Size                  2230952 bytes
Variable Size             222299480 bytes
Database Buffers          394264576 bytes
Redo Buffers                7532544 bytes
Database mounted.
Database opened.

发现HAS的服务确实启动了的,而ora.cssd和ora.diskmon这2个服务是依赖于HAS维护的.

注意：启动ASM实例用：sqlplus "/as sysasm"

1)默认情况下HAS(High Availability Service)是自动启动的.通过如下命令可以取消和启用自动启动
        crsctl disable has
       crsctl enable has
2)HAS手动启动和停止
        crsctl start has
       crsctl stop has
3)查看HAS的状态
        crsctl check has
4)如果想让ora.css和ora.diskmon服务随着HAS的启动而自动启动，那么你可以这两个服务的AUTO_START属性
        crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
        or 
        crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"
5)如果想取消ora.css和ora.diskmon的Auto start 
        crsctl modify resource "ora.cssd" -attr "AUTO_START=never"
        crsctl modify resource "ora.diskmon" -attr "AUTO_START=never"