PRKH-1010 :Unable to communicate with CRS services.CRS-4535: Cannot communicate with Cluster Ready S

好久不开机的服务器开机后，rac没有自动启动。

[oracle@testpdb1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期二 10月 29 14:10:00 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> exit
已断开连接

[oracle@testpdb1 ~]$ srvctl status nodeapps -n testpdb1
PRKH-1010 : Unable to communicate with CRS services.
PRKH-3003 : An attempt to communicate with the CSS daemon failed
[oracle@testpdb1 ~]$ crs_stat -t
bash: crs_stat: command not found...
[oracle@testpdb1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 29-10月-2019 14:12:05

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

正在连接到 (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS: 无监听程序
TNS-12560: TNS: 协议适配器错误
  TNS-00511: 无监听程序
   Linux Error: 111: Connection refused

[grid@testpdb1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@testpdb1 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@testpdb1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]

check crs的时候只出现crs-4535的错误，解决步骤：
这个问题一般是由crsd进程死掉或者ocr磁盘组状态不对引起的。
1.查看磁盘组所属组(chown)和权限(chmod)是否正确
2.查看crsd的进程是否存在或者正确
ps -ef | grep crsd.bin
crsctl stat res -t -init 查看ora.crsd的状态是否为online

[grid@testpdb1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE                                                   
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE                                                   
ora.crf
      1        ONLINE  ONLINE       testpdb1                                     
ora.crsd
      1        ONLINE  OFFLINE                                                   
ora.cssd
      1        ONLINE  OFFLINE                                                   
ora.cssdmonitor
      1        ONLINE  ONLINE       testpdb1                                     
ora.ctssd
      1        ONLINE  OFFLINE                                                   
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  OFFLINE                                                   
ora.gipcd
      1        ONLINE  ONLINE       testpdb1                                     
ora.gpnpd
      1        ONLINE  ONLINE       testpdb1                                     
ora.mdnsd
      1        ONLINE  ONLINE       testpdb1          

3.一般到这一步其实ora.crsd的状态是offline的，所以需要去检查一下grid 下的$ORACLE_HOME/log/hostname/crsd/crsd.log日志的输出，看是否存在异常
4.检查asm磁盘组的状态
select name,state from v$asm_diskgroup;想对应的ocr磁盘组的状态
5.在所有都检查完毕后没有问题，就启动crsd进程

[grid@testpdb1 ~]$ crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.cssd' on 'testpdb1'
CRS-2672: Attempting to start 'ora.diskmon' on 'testpdb1'
CRS-2676: Start of 'ora.diskmon' on 'testpdb1' succeeded
CRS-2676: Start of 'ora.cssd' on 'testpdb1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'testpdb1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'testpdb1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'testpdb1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'testpdb1'
CRS-2676: Start of 'ora.ctssd' on 'testpdb1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'testpdb1'
CRS-2676: Start of 'ora.evmd' on 'testpdb1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'testpdb1' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'testpdb1'
CRS-2681: Clean of 'ora.asm' on 'testpdb1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'testpdb1'
CRS-2676: Start of 'ora.asm' on 'testpdb1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'testpdb1'
CRS-2676: Start of 'ora.crsd' on 'testpdb1' succeeded
[grid@testpdb1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       testpdb1                 Started             
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       testpdb1                                     
ora.crf
      1        ONLINE  ONLINE       testpdb1                                     
ora.crsd
      1        ONLINE  ONLINE       testpdb1                                     
ora.cssd
      1        ONLINE  ONLINE       testpdb1                                     
ora.cssdmonitor
      1        ONLINE  ONLINE       testpdb1                                     
ora.ctssd
      1        ONLINE  ONLINE       testpdb1                 ACTIVE:0            
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  ONLINE       testpdb1                                     
ora.gipcd
      1        ONLINE  ONLINE       testpdb1                                     
ora.gpnpd
      1        ONLINE  ONLINE       testpdb1                                     
ora.mdnsd
      1        ONLINE  ONLINE       testpdb1                                     
[grid@testpdb1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@testpdb1 ~]$ crsctl check cluster
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online


[oracle@testpdb1 ~]$ lsnrctl stat

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 29-10月-2019 14:54:05

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

正在连接到 (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
LISTENER 的 STATUS
------------------------
别名                      LISTENER
版本                      TNSLSNR for Linux: Version 11.2.0.4.0 - Production
启动日期                  29-10月-2019 14:14:07
正常运行时间              0 天 0 小时 39 分 58 秒
跟踪级别                  off
安全性                    ON: Local OS Authentication
SNMP                      OFF
监听程序日志文件          /u01/app/oracle/diag/tnslsnr/testpdb1/listener/alert/log.xml
监听端点概要...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=testpdb1.nsfc.gov.cn)(PORT=1521)))
服务摘要..
服务 "+ASM" 包含 1 个实例。
  实例 "+ASM1", 状态 READY, 包含此服务的 1 个处理程序...
服务 "racpdb" 包含 1 个实例。
  实例 "racpdb1", 状态 READY, 包含此服务的 2 个处理程序...
服务 "racpdb.nsfc.gov.cn" 包含 1 个实例。
  实例 "racpdb1", 状态 READY, 包含此服务的 2 个处理程序...
服务 "racpdbXDB" 包含 1 个实例。
  实例 "racpdb1", 状态 READY, 包含此服务的 1 个处理程序...
命令执行成功