两节点时间不同步导致CRS-0184

节点二：
[oracle@rac2 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon

节点一：
[root@rac1 ~]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1
ora.FRA.dg     ora....up.type ONLINE    ONLINE    rac1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1
ora....TING.dg ora....up.type ONLINE    ONLINE    rac1
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac1
ora.dave.db    ora....se.type ONLINE    ONLINE    rac1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    rac1
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac1
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac1
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1

查看节点二alert日志：

 tail -150f alertrac2.log | less
2013-08-17 09:26:08.534
[ctssd(2257)]CRS-2405:The Cluster Time Synchronization Service on host rac2 is shutdown by user
2013-08-17 09:26:17.886
[ctssd(2265)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
2013-08-17 09:26:17.887
[ctssd(2265)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2013-08-17 09:26:22.222
[ctssd(2265)]CRS-2404:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/grid/11.2.0/log/rac2/ctssd/octssd.log.
2013-08-17 09:26:22.493
[ctssd(2265)]CRS-2402:The Cluster Time Synchronization Service aborted on host rac2. Details at  in /u01/app/grid/11.2.0/log/rac2/ctssd/octssd.log.
2013-08-17 09:26:34.347
[ctssd(2284)]CRS-2401:The Cluster Time Synchronization Service started on host rac2.
2013-08-17 09:26:34.347
[ctssd(2284)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2013-08-17 09:26:52.911

节点二octssd.log日志：
[root@rac2 cssd]# tail -f /u01/app/grid/11.2.0/log/rac2/ctssd/octssd.log
2013-08-17 10:10:23.046: [    CTSS][1231296256]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm2_3: Received time sync message from master.
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm: The magnitude [37935421] of the offset [37935421 usec] is larger than [10000000 usec] which is the CTSS limit that can be synced up in a short period.
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm: The magnitude of the systime diff is larger than max adjtime limit. Offset [37935421] usec will be changed to max adjtime limit [+/- 131071].
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm15: The CTSS master is behind this node. The local time offset [-131071 usec] is being adjusted. Sync method [2]
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm17: LT [1376705423sec 46690usec], MT [1376705385sec 140716013630117usec], Delta [14739usec]
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm19: The offset is [131071 usec] and sync interval set to [4]
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm: Received from master (mode [0xcc] nodenum [1] hostname [rac1] )
2013-08-17 10:10:23.047: [    CTSS][1231296256]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler
2013-08-17 10:10:23.047: [    CTSS][1227093760]ctsselect_msm: Sync interval returned in [4]
2013-08-17 10:10:24.056: [    CTSS][1227093760]ctsselect_msm: CTSS mode is [0xc4]
2013-08-17 10:10:24.057: [    CTSS][1227093760]ctssslave_swm1_2: Ready to initiate new time sync process.
2013-08-17 10:10:24.059: [    CTSS][1227093760]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].
2013-08-17 10:10:24.070: [    CTSS][1231296256]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].
2013-08-17 10:10:24.070: [    CTSS][1231296256]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].
2013-08-17 10:10:24.070: [    CTSS][1227093760]ctssslave_swm2_3: Received time sync message from master.

从日志看两个节点时间不同步，时间差超过CTSS的限制。

2013-08-17 10:10:23.047: [    CTSS][1227093760]ctssslave_swm: The magnitude [37935421] of the offset [37935421 usec] is larger than [10000000 usec] which is the CTSS limit that can be synced up in a short period.

SQL> select 10000000/1000000 from dual;

10000000/1000000
----------------
              10

SQL> select 37935421/1000000 from dual;

37935421/1000000
----------------
       37.935421

 

同步两节点时间后：

[root@rac2 ~]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1
ora.FRA.dg     ora....up.type ONLINE    ONLINE    rac1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1
ora....TING.dg ora....up.type ONLINE    ONLINE    rac1
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac1
ora.dave.db    ora....se.type ONLINE    ONLINE    rac1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    rac1
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac1
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1
ora....SM2.asm application    ONLINE    ONLINE    rac2
ora....C2.lsnr application    ONLINE    ONLINE    rac2
ora.rac2.gsd   application    OFFLINE   OFFLINE
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1