19C RAC打补丁的时候虚拟机突然断开了，重启后节点一集群无法启动

zongzizz

于 2023-07-02 19:11:04 发布

阅读量346

点赞数

分类专栏： Oracle数据库基础 RAC 补丁文章标签：数据库 oracle

本文链接：https://blog.csdn.net/zongzizz/article/details/131503943

版权

Oracle数据库基础同时被 3 个专栏收录

54 篇文章 1 订阅

订阅专栏

RAC

11 篇文章 0 订阅

订阅专栏

补丁

1 篇文章 0 订阅

订阅专栏

在尝试将两个19.3版RAC节点升级到19.14时，由于磁盘空间不足，打补丁过程中导致节点卡死并重启。节点一无法启动集群，日志显示SCAN名称解析不一致，只解析到一个IP地址，且OracleHome下的库存文件可能损坏。节点二运行正常。解决方法包括完成补丁设置，锁定GIhome并重新启动GI。

摘要由CSDN通过智能技术生成

RAC两个节点19.3的版本，准备打补丁升级到19.14，在打节点1的grid补丁的时候两个节点虚拟机因为电脑磁盘空间不够，打补丁时候导致电脑磁盘空间涨死，导致两个节点的主机都卡死断开连接。清理空间后重启两节点，节点一集群启动不了，节点2正常。crs日志如下：

2023-07-02 15:54:39.446 [CVUD(5047)]CRS-10051: CVU found following errors with Clusterware setup : PRVF-4664 : Found inconsistent name resolution entries for SCAN name "xyzdb-scan"
PRVG-11368 : A SCAN is recommended to resolve to "3" or more IP addresses, but SCAN "xyzdb-scan" resolves to only "/192.168.0.20"
PRVG-1260 : Command "/u01/app/19.3.0.0/grid/cv/remenv/cvuhelper /u01/app/19.3.0.0/grid 19 /u01/app/19.3.0.0/grid/jlib/cvuhelper19.jar /u01/app/19.3.0.0/grid -getOraclePatchList /u01/app/19.3.0.0/grid" to obtain Oracle patch status failed
Unable to create patchObject,Possible causes are:,ORACLE_HOME/inventory/oneoffs/29401763_en_22759421 is corrupted. java.lang.RuntimeException: No Patch exists,Please check.,,

2023-07-02 15:54:39.446 [CVUD(5047)]CRS-10051: CVU found following errors with Clusterware setup : PRVG-10466 : Failed to determine cluster node roles. Verification will proceed considering nodes "xyzdb1,xyzdb2" as hub nodes.
PRCI-1108 : Failed to check CRS running state for CRS home /u01/app/19.3.0.0/grid on node xyzdb1
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
PRVG-10466 : Failed to determine cluster node roles. Verification will proceed considering nodes "xyzdb1,xyzdb2" as hub nodes.
PRCI-1108 : Failed to check CRS running state for CRS home /u01/app/19.3.0.0/grid on node xyzdb1
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

2023-07-02 16:38:40.262 [CRSD(4311)]CRS-2773: Server 'xyzdb1' has been removed from pool 'Free'.
2023-07-02 16:38:40.262 [CRSD(4311)]CRS-2772: Server 'xyzdb1' has been assigned to pool 'Generic'.
2023-07-02 16:38:40.263 [CRSD(4311)]CRS-2773: Server 'xyzdb2' has been removed from pool 'Free'.
2023-07-02 16:38:40.263 [CRSD(4311)]CRS-2772: Server 'xyzdb2' has been assigned to pool 'Generic'.
2023-07-02 16:38:40.264 [CRSD(4311)]CRS-2772: Server 'xyzdb1' has been assigned to pool 'ora.orcl'.
2023-07-02 16:38:40.264 [CRSD(4311)]CRS-2772: Server 'xyzdb2' has been assigned to pool 'ora.orcl'.
2023-07-02 16:50:20.568 [ORAAGENT(28324)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 28324
2023-07-02 17:37:49.456 [OHASD(2270)]CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'xyzdb1'
2023-07-02 17:38:06.795 [ORAROOTAGENT(4524)]CRS-5822: Agent '/u01/app/19.3.0.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:3:112} in /u01/app/grid/diag/crs/xyzdb1/crs/trace/crsd_orarootagent_root.trc.
2023-07-02 17:38:09.599 [MDNSD(3131)]CRS-5602: mDNS service stopping by request.
2023-07-02 17:38:10.235 [MDNSD(3131)]CRS-8504: Oracle Clusterware MDNSD process with operating system process ID 3131 is exiting
2023-07-02 17:38:26.459 [OCTSSD(3927)]CRS-2405: The Cluster Time Synchronization Service on host xyzdb1 is shutdown by user
2023-07-02 17:38:26.477 [OCTSSD(3927)]CRS-8504: Oracle Clusterware OCTSSD process with operating system process ID 3927 is exiting
2023-07-02 17:38:27.493 [OCSSD(3538)]CRS-1603: CSSD on node xyzdb1 has been shut down.
2023-07-02 17:38:30.502 [GPNPD(3217)]CRS-2329: GPNPD on node xyzdb1 shut down. 
2023-07-02 17:38:31.515 [OHASD(2270)]CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'xyzdb1' has completed
2023-07-02 17:38:31.525 [ORAROOTAGENT(2908)]CRS-5822: Agent '/u01/app/19.3.0.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:2:12} in /u01/app/grid/diag/crs/xyzdb1/crs/trace/ohasd_orarootagent_root.trc.

尝试手动启动节点1crs，报错如下：

[root@xyzdb1 bin]# ./crsctl start crs
CRS-6706: Oracle Clusterware Release patch level ('506409770') does not match Software patch level ('724960844'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

都是指向打补丁失败的原因。

解决方法参考Patching 12.2.0.1 Grid Infrastructure gives error CRS-6706: Oracle Clusterware Release Patch Level (‘748994161’) Does Not Match Software Patch Level (Doc ID 2348013.1)

1.  Run the following command as the root user to complete the patching set up behind the scenes:

#GI_HOME/bin:>  ./clscfg -localpatch


2.  Run the following command as the root user to lock the GI home:

#GI_HOME/crs/install:>  ./rootcrs.sh -lock

 
3.  Run the following command as the root user to start the GI:

#GI_HOME/bin:>  ./crsctl start crs