CRS-0184: Cannot communicate with the CRS daemon.

 今天遇到一个环境，某节点无法运行crs_stat相关命令

$ crs_stat -t                 CRS-0184: Cannot communicate with the CRS daemon.                                 $ crsctl check has                 CRS-4639: Could not contact Oracle High Availability Services

在这个节点查看IP地址，VIP已经不存在

$ ifconfig -a                 en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>                         inet 192.168.0.70 netmask 0xffffff00 broadcast 192.168.0.255                          tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1                 en1: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>                         inet 10.10.10.70 netmask 0xff000000 broadcast 10.255.255.255                          tcp_sendspace 65536 tcp_recvspace 65536 rfc1323 1                 lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>                         inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255                         inet6 ::1%1/0                          tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

于是在其他节点发现，节点1好像已经退出集群环境

# su - grid $ crs_stat -t Name           Type           Target    State     Host         ------------------------------------------------------------ ora.DATA.dg    ora....up.type ONLINE    ONLINE    node2        ora....ER.lsnr ora....er.type ONLINE    ONLINE    node2        ora....N1.lsnr ora....er.type ONLINE    OFFLINE                ora.OCR.dg     ora....up.type ONLINE    ONLINE    node2        ora.asm        ora.asm.type   ONLINE    ONLINE    node2        ora.cvu        ora.cvu.type   ONLINE    OFFLINE                ora.gsd        ora.gsd.type   OFFLINE   OFFLINE                ora....network ora....rk.type ONLINE    ONLINE    node2        ora.node1.vip  ora....t1.type ONLINE    OFFLINE                ora....SM2.asm application    ONLINE    ONLINE    node2        ora....E2.lsnr application    ONLINE    ONLINE    node2        ora.node2.gsd  application    OFFLINE   OFFLINE                ora.node2.ons  application    ONLINE    ONLINE    node2        ora.node2.vip  ora....t1.type ONLINE    ONLINE    node2        ora.oc4j       ora.oc4j.type  ONLINE    OFFLINE                ora.ons        ora.ons.type   ONLINE    ONLINE    node2        ora.prod.db    ora....se.type ONLINE    ONLINE    node2        ora....ry.acfs ora....fs.type ONLINE    ONLINE    node2        ora.scan1.vip  ora....ip.type ONLINE    OFFLINE

而且节点1的VIP已经飘逸到节点2

$ ifconfig -a en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>         inet 192.168.0.75 netmask 0xffffff00 broadcast 192.168.0.255         inet 192.168.0.79 netmask 0xffffff00 broadcast 192.168.0.255         inet 192.168.0.74 netmask 0xffffff00 broadcast 192.168.0.255          tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1 en1: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>         inet 10.10.10.75 netmask 0xff000000 broadcast 10.255.255.255         inet 169.254.131.177 netmask 0xffff0000 broadcast 169.254.255.255          tcp_sendspace 65536 tcp_recvspace 65536 rfc1323 1 lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>         inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255         inet6 ::1%1/0          tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

于是清理节点1的配置信息，打算重新运行root.sh

# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -verbose -deconfig -force  Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type PRCR-1068 : Failed to query resources Cannot communicate with crsd PRCR-1070 : Failed to check if resource ora.gsd is registered Cannot communicate with crsd PRCR-1070 : Failed to check if resource ora.ons is registered Cannot communicate with crsd CRS-4535: Cannot communicate with Cluster Ready Services CRS-4000: Command Stop failed, or completed with errors. CRS-4544: Unable to connect to OHAS CRS-4000: Command Stop failed, or completed with errors. ACFS-9313: No ADVM/ACFS installation detected. Successfully deconfigured Oracle clusterware stack on this node # # # /u01/app/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params CRS-4047: No Oracle Clusterware components configured. CRS-4000: Command Stop failed, or completed with errors. CRS-4047: No Oracle Clusterware components configured. CRS-4000: Command Delete failed, or completed with errors. CRS-4047: No Oracle Clusterware components configured. CRS-4000: Command Stop failed, or completed with errors. You must kill ohasd processes or reboot the system to properly  cleanup the processes started by Oracle clusterware ACFS-9313: No ADVM/ACFS installation detected. Either /etc/oracle/olr.loc does not exist or is not readable Make sure the file exists and it has read and execute access Failure in execution (rc=-1, 512, A file or directory in the path name does not exist.) for command /etc/ohasd deinstall Successfully deconfigured Oracle Restart stack

环境清理后，运行root.sh

# cd /u01/app/11.2.0/grid/ #  # ls JRE             crs             diagnostics     instantclient   md              oraInst.loc     racg            sysman OPatch          csmig           eons            inventory       mdns            oracore         rdbms           usm assistants      css             evm             javavm          mesg            ord             relnotes        utl auth            ctss            gipc            jdbc            network         osysmond        root.sh         wwg bin             cv              gns             jdk             nls             oui             rootupgrade.sh  xdk cdata           dbs             gpnp            jlib            oc4j            owm             scheduler cfgtoollogs     dc_ocm          has             ldap            ohasd           perl            slax clone           deinstall       hs              lib             ologgerd        plsql           sqlplus crf             demo            install         log             opmn            precomp         srvm #  #  # ./root.sh Performing root user operation for Oracle 11g  The following environment variables are set as:     ORACLE_OWNER= grid     ORACLE_HOME=  /u01/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]:  The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params User ignored Prerequisites during installation User grid has the required capabilities to run CSSD in realtime mode OLR initialization - successful Adding Clusterware entries to inittab CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node2, number 2, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster PRKO-2190 : VIP exists for node node1, VIP name node1-vip Configure Oracle Grid Infrastructure for a Cluster ... succeeded

查看结果：

$ crs_stat -t Name           Type           Target    State     Host         ------------------------------------------------------------ ora.DATA.dg    ora....up.type ONLINE    ONLINE    node1        ora....ER.lsnr ora....er.type ONLINE    ONLINE    node1        ora....N1.lsnr ora....er.type ONLINE    ONLINE    node1        ora.OCR.dg     ora....up.type ONLINE    ONLINE    node1        ora.asm        ora.asm.type   ONLINE    ONLINE    node1        ora.cvu        ora.cvu.type   ONLINE    ONLINE    node2        ora.gsd        ora.gsd.type   OFFLINE   OFFLINE                ora....network ora....rk.type ONLINE    ONLINE    node1        ora....SM1.asm application    ONLINE    ONLINE    node1        ora....E1.lsnr application    ONLINE    ONLINE    node1        ora.node1.gsd  application    OFFLINE   OFFLINE                ora.node1.ons  application    ONLINE    ONLINE    node1        ora.node1.vip  ora....t1.type ONLINE    ONLINE    node1        ora....SM2.asm application    ONLINE    ONLINE    node2        ora....E2.lsnr application    ONLINE    ONLINE    node2        ora.node2.gsd  application    OFFLINE   OFFLINE                ora.node2.ons  application    ONLINE    ONLINE    node2        ora.node2.vip  ora....t1.type ONLINE    ONLINE    node2        ora.oc4j       ora.oc4j.type  ONLINE    OFFLINE                ora.ons        ora.ons.type   ONLINE    ONLINE    node1        ora.prod.db    ora....se.type ONLINE    ONLINE    node1        ora....ry.acfs ora....fs.type ONLINE    ONLINE    node1        ora.scan1.vip  ora....ip.type ONLINE    ONLINE    node1