19c集群 两节点时间相差太大导致集群异常

客户反馈集群有故障了，有个节点无法启动，登录查看集群的alert.log日志，发现一直报

2023-10-17 11:04:12.260 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 11:34:12.975 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 12:04:13.669 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 12:34:14.364 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 13:04:15.065 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 13:34:15.800 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 14:04:16.543 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 14:34:17.298 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 15:04:18.037 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 15:34:18.760 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 16:04:19.510 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 16:34:20.255 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 17:04:20.986 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 17:34:21.723 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 18:04:22.465 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 18:34:23.194 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 19:04:23.920 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 19:34:24.635 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-17 20:04:25.372 [OCTSSD(22948)]CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.
 Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.

.........................

.........................

2023-10-19 16:50:41.165 [OCTSSD(5435)]CRS-2419: The clock on host db1 differs from mean cluster time by 1199033595 microseconds. The Cluster Time Synchronization Service wi
ll not perform time synchronization because the time difference is beyond the permissible offset of 600 seconds. Details in /u01/app/grid/diag/crs/db1/crs/trace/octssd.trc.
2023-10-19 16:50:41.766 [OCTSSD(5435)]CRS-2402: The Cluster Time Synchronization Service aborted on host db1. Details at (:ctsselect_msm3:) in /u01/app/grid/diag/crs/db1/cr
s/trace/octssd.trc.
2023-10-26 18:33:08.168 [OHASD(3226)]CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'db1'
2023-10-26 18:33:10.132 [MDNSD(4238)]CRS-5602: mDNS service stopping by request.
2023-10-26 18:33:10.742 [MDNSD(4238)]CRS-8504: Oracle Clusterware MDNSD process with operating system process ID 4238 is exiting
2023-10-26 18:33:11.168 [OCSSD(5173)]CRS-1603: CSSD on node db1 has been shut down.
2023-10-26 18:33:14.176 [GPNPD(4353)]CRS-2329: GPNPD on node db1 shut down.
2023-10-26 18:33:16.204 [OHASD(3226)]CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'db1' has completed
2023-10-26 18:33:16.218 [ORAROOTAGENT(3877)]CRS-5822: Agent '/u01/app/19.0.0/grid_1/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:4:11} in
/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc.
2023-10-26 18:38:05.468 [OHASD(3058)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 3058
2023-10-26 18:38:05.625 [OHASD(3058)]CRS-0714: Oracle Clusterware Release 19.0.0.0.0.
2023-10-26 18:38:05.660 [OHASD(3058)]CRS-2112: The OLR service started on node db1.
2023-10-26 18:38:06.088 [OHASD(3058)]CRS-1301: Oracle High Availability Service started on node db1.
2023-10-26 18:38:06.141 [OHASD(3058)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2023-10-26 18:38:07.627 [ORAROOTAGENT(3688)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 3688
2023-10-26 18:38:07.946 [CSSDMONITOR(3704)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 3704
2023-10-26 18:38:07.946 [CSSDAGENT(3700)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 3700
2023-10-26 18:38:07.958 [ORAAGENT(3698)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 3698
2023-10-26 18:38:08.837 [ORAROOTAGENT(3688)]CRS-5016: Process "/u01/app/19.0.0/grid_1/bin/acfsload" spawned by agent "ORAROOTAGENT" for action "check" failed: details at "(
:CLSN00010:)" in "/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc"
2023-10-26 18:38:08.753 [ORAAGENT(3827)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 3827
2023-10-26 18:38:09.214 [MDNSD(3882)]CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 3882
2023-10-26 18:38:09.176 [CLSECHO(3929)]ACFS-9391: Checking for existing ADVM/ACFS installation.
2023-10-26 18:38:09.263 [EVMD(3880)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 3880
2023-10-26 18:38:09.784 [CLSECHO(3945)]ACFS-9392: Validating ADVM/ACFS installation files for operating system.
2023-10-26 18:38:09.812 [CLSECHO(3953)]ACFS-9393: Verifying ASM Administrator setup.
2023-10-26 18:38:09.873 [CLSECHO(3964)]ACFS-9308: Loading installed ADVM/ACFS drivers.
2023-10-26 18:38:10.255 [GPNPD(3985)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 3985
2023-10-26 18:38:11.098 [GPNPD(3985)]CRS-2328: GPNPD started on node db1.
2023-10-26 18:38:11.239 [GIPCD(4126)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 4126
2023-10-26 18:38:11.770 [CLSECHO(4207)]ACFS-9154: Loading 'oracleoks.ko' driver.
2023-10-26 18:38:12.582 [CLSECHO(4283)]ACFS-9154: Loading 'oracleadvm.ko' driver.
2023-10-26 18:38:13.300 [CLSECHO(4434)]ACFS-9154: Loading 'oracleacfs.ko' driver.
2023-10-26 18:38:15.366 [CLSECHO(4617)]CRS-10001: ACFS-9325:     Driver OS kernel version = 4.14.35-1902.0.9.el7uek.x86_64.

看日志应该是两节点时间差太大，查看侯发现相差20分钟，

+ASM1:/home/grid@db1> ssh db2 date; date
Fri Oct 27 14:37:26 CST 2023
Fri Oct 27 14:57:32 CST 2023
+ASM1:/home/grid@db1>

因等保原因，服务器和时钟源网络断了。

首先手动调整时间后，手动启动db1的crs服务，启动正常，实例也自动恢复。

等网络负责人调整好网络再查看时钟同步