Oracle 19c RAC升级补丁19.14报错记录

环境：linux7.6+Oracle19c RAC
问题描述：数据新环境部署，从19.3直接升级到19.14，期间grid打补丁成功，但db失败，失败日志如下：

[root@cdjkdb1 patch]# /oracle/app/oracle/product/19.3.0/db_1/OPatch/opatchauto apply /oracle/patch/33567274/33509923 -oh /oracle/app/oracle/product/19.3.0/db_1

OPatchauto session is initiated at Thu Oct 13 20:46:51 2022


System initialization log file is /oracle/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchautodb/systemconfig2022-10-13_08-46-53PM.log.

Session log file is /oracle/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/opatchauto2022-10-13_08-47-14PM.log
The id for this session is 393P

Executing OPatch prereq operations to verify patch applicability on home /oracle/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /oracle/app/oracle/product/19.3.0/db_1


Executing patch validation checks on home /oracle/app/oracle/product/19.3.0/db_1
Patch validation checks successfully completed on home /oracle/app/oracle/product/19.3.0/db_1


Verifying SQL patch applicability on home /oracle/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /oracle/app/oracle/product/19.3.0/db_1


Preparing to bring down database service on home /oracle/app/oracle/product/19.3.0/db_1
Successfully prepared home /oracle/app/oracle/product/19.3.0/db_1 to bring down database service


Bringing down database service on home /oracle/app/oracle/product/19.3.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: mochabsm
Database service successfully brought down on home /oracle/app/oracle/product/19.3.0/db_1


Performing prepatch operation on home /oracle/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /oracle/app/oracle/product/19.3.0/db_1


Start applying binary patch on home /oracle/app/oracle/product/19.3.0/db_1
Failed while applying binary patches on home /oracle/app/oracle/product/19.3.0/db_1

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : cdjkdb1->/oracle/app/oracle/product/19.3.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /oracle/app/oracle/product/19.3.0/db_1, host: cdjkdb1.
Command failed:  /oracle/app/oracle/product/19.3.0/db_1/OPatch/opatchauto  apply /oracle/patch/33567274/33509923 -oh /oracle/app/oracle/product/19.3.0/db_1 -target_type rac_database -binary -invPtrLoc /oracle/app/oracle/product/19.3.0/db_1/oraInst.loc -jre /oracle/app/oracle/product/19.3.0/db_1/OPatch/jre -persistresult /oracle/app/oracle/product/19.3.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_cdjkdb1_rac_2.ser -analyzedresult /oracle/app/oracle/product/19.3.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_cdjkdb1_rac_2.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /oracle/patch/33567274/33509923/33515361
Log: /oracle/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-10-13_20-49-00PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed. 

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Thu Oct 13 20:49:54 2022
Time taken to complete the session 3 minutes, 3 seconds

 opatchauto failed with error code 42

最初由于忽略了日志的info输出信息，在网上以及mos上找各种方式尝试，均已失败告终，随后重新研究日志信息，定位了原因，最终顺利升级。
错误方式一：
排除oui-patch.xml文件权限，日志里没有这一项，排查无意义；
错误方式二：
排查oneoff目录下的内容，日志里也没有这一项。
正确方式：
根据日志提示信息，有一个关键项：libxxx.so（详见下图）正在使用，不能被opatch使用的信息，使用fuser libxxx.so查看后，会有显示占用的进程号，

kill -9 进程号
结束进程，再执行就顺利完成了。
值得注意的是，其实在补丁升级的过程中报错信息在日志里已经有明显提示了，所以还是先仔细看日志，然后再根据日志提示去针对性的处理。