oracle 12c chad,Oracle12C集群打补丁失败转而使用手工打补丁报CRS-6706处理过程

Oracle12c 12.2.0.1 opatch auto 打集群补丁失败，报错信息如下

Following active excutables are used by opatch process:

/oracle/12.2/lib/libclntsh.so.12.1

/oracle/12.2/lib/libsqlplus.so

Prerequisite check "CheckActiveFileAndExecutables" failed

如下查询没有任何输出

fuser /oracle/12.2/lib/libclntsh.so.12.1

fuser /oracle/12.2/lib/libsqlplus.so

我们尝试使用手工方式打集群和数据库补丁

opatch apply /patch-number -oh

opatch apply /patch-number -oh

成功！

然后尝试重启集群，报错如下

CRS-6706: Oracle Clusterware Release patch level('2197668489') does not match software opatch level('0').Oracle Clusterware cannot be started

CRS-4000: Command Start failed,or completed with errors.

这个错误比较明显是patch level不匹配，于是我们找到如下文章

CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm') (Doc ID 1639285.1)

vThis note lists causes/solutions for the following issue:

Oracle 12c Grid Infrastructure fails to start after patching:

CRS-6706: Oracle Clusterware Release patch level ('xxxxxxxx') does not match Software patch level ('xxxxxxxxx'). Oracle Clusterware cannot be started

This can be encountered - for example - in cases where the post patch application instruction to run 'rootcrs.pl -postpatch' is not followed.

由于 其他节点使用opatchauto打补丁，集群操作打补丁都自动完成，无需执行rootcrs.pl -postpatch操作，而此时，我们改为手工打补丁，所以忽略了这个步骤。

问题解决：

[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.sh -prepatch

Using configuration parameter file: /grid/12.2/crs/install/crsconfig_params

The log of current session can be found at:

/grid/base/crsdata/pc156/crsconfig/crspatch_pc156_2021-01-29_02-51-04PM.log

[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.pl -postpatch

Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /grid/12.2/crs/install /grid/12.2/crs/install/../../perl/lib) at /grid/12.2/crs/install/crsinstall.pm line 296.

BEGIN failed--compilation aborted at /grid/12.2/crs/install/crsinstall.pm line 296.

Compilation failed in require at /grid/12.2/crs/install/rootcrs.pl line 165.

BEGIN failed--compilation aborted at /grid/12.2/crs/install/rootcrs.pl line 165.

这里rootcrs.pl -postpatch操作失败，提示也比较明显缺少Env.pm，下面安装组件

[root@n-pc-sr850-308 ~]#yum -y install perl-Env

Loaded plugins: ulninfo

Resolving Dependencies

--> Running transaction check

---> Package perl-Env.noarch 0:1.04-2.el7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================

Package Arch Version Repository Size

============================================================================================================================

Installing:

perl-Env noarch 1.04-2.el7 ol74 15 k

Transaction Summary

============================================================================================================================

Install 1 Package

Total download size: 15 k

Installed size: 27 k

Downloading packages:

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : perl-Env-1.04-2.el7.noarch 1/1

Verifying : perl-Env-1.04-2.el7.noarch 1/1

Installed:

perl-Env.noarch 0:1.04-2.el7

Complete!

继续执行[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.pl -postpatch

[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.pl -postpatch

Using configuration parameter file: /grid/12.2/crs/install/crsconfig_params

The log of current session can be found at:

/grid/base/crsdata/pc156/crsconfig/crspatch_pc156_2021-01-29_02-57-17PM.log

2021/01/29 14:57:22 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2021/01/29 14:58:35 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2021/01/29 14:58:37 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.evmd' on 'pc156'

CRS-2672: Attempting to start 'ora.mdnsd' on 'pc156'

CRS-2676: Start of 'ora.mdnsd' on 'pc156' succeeded

CRS-2676: Start of 'ora.evmd' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'pc156'

CRS-2676: Start of 'ora.gpnpd' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'pc156'

CRS-2676: Start of 'ora.gipcd' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'pc156'

CRS-2676: Start of 'ora.cssdmonitor' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'pc156'

CRS-2672: Attempting to start 'ora.diskmon' on 'pc156'

CRS-2676: Start of 'ora.diskmon' on 'pc156' succeeded

CRS-2676: Start of 'ora.cssd' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'pc156'

CRS-2672: Attempting to start 'ora.ctssd' on 'pc156'

CRS-2676: Start of 'ora.ctssd' on 'pc156' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'pc156'

CRS-2676: Start of 'ora.asm' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'pc156'

CRS-2676: Start of 'ora.storage' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'pc156'

CRS-2676: Start of 'ora.crf' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'pc156'

CRS-2676: Start of 'ora.crsd' on 'pc156' succeeded

CRS-6023: Starting Oracle Cluster Ready Services-managed resources

CRS-6017: Processing resource auto-start for servers: pc156

CRS-2673: Attempting to stop 'ora.pc156.vip' on 'n-pc-sr850-322'

CRS-2672: Attempting to start 'ora.ons' on 'pc156'

CRS-2672: Attempting to start 'ora.chad' on 'pc156'

CRS-2677: Stop of 'ora.pc156.vip' on 'n-pc-sr850-322' succeeded

CRS-2672: Attempting to start 'ora.pc156.vip' on 'pc156'

CRS-2676: Start of 'ora.chad' on 'pc156' succeeded

CRS-2676: Start of 'ora.pc156.vip' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'pc156'

CRS-2676: Start of 'ora.ons' on 'pc156' succeeded

CRS-2676: Start of 'ora.LISTENER.lsnr' on 'pc156' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'pc156'

CRS-2676: Start of 'ora.asm' on 'pc156' succeeded

CRS-6016: Resource auto-start has completed for server pc156

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [2197668489].

集群启动成功！