环境:linux7.6+Oracle19c RAC
问题描述:数据新环境部署,从19.3直接升级到19.14,期间grid打补丁成功,但db失败,失败日志如下:
[root@cdjkdb1 patch]# /oracle/app/oracle/product/19.3.0/db_1/OPatch/opatchauto apply /oracle/patch/33567274/33509923 -oh /oracle/app/oracle/product/19.3.0/db_1
OPatchauto session is initiated at Thu Oct 13 20:46:51 2022
System initialization log file is /oracle/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchautodb/systemconfig2022-10-13_08-46-53PM.log.
Session log file is /oracle/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/opatchauto2022-10-13_08-47-14PM.log
The id for this session is 393P
Executing OPatch prereq operations to verify patch applicability on home /oracle/app/oracle/product/19.3.0/db_1
Patch applicability verified successfully on home /oracle/app/oracle/product/19.3.0/db_1
Executing patch validation checks on home /oracle/app/oracle/product/19.3.0/db_1
Patch validation checks successfully completed on home /oracle/app/oracle/product/19.3.0/db_1
Verifying SQL patch applicability on home /oracle/app/oracle/product/19.3.0/db_1
SQL patch applicability verified successfully on home /oracle/app/oracle/product/19.3.0/db_1
Preparing to bring down database service on home /oracle/app/oracle/product/19.3.0/db_1
Successfully prepared home /oracle/app/oracle/product/19.3.0/db_1 to bring down database service
Bringing down database service on home /oracle/app/oracle/product/19.3.0/db_1
Following database(s) and/or service(s) are stopped and will be restarted later during the session: mochabsm
Database service successfully brought down on home /oracle/app/oracle/product/19.3.0/db_1
Performing prepatch operation on home /oracle/app/oracle/product/19.3.0/db_1
Perpatch operation completed successfully on home /oracle/app/oracle/product/19.3.0/db_1
Start applying binary patch on home /oracle/app/oracle/product/19.3.0/db_1
Failed while applying binary patches on home /oracle/app/oracle/product/19.3.0/db_1
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : cdjkdb1->/oracle/app/oracle/product/19.3.0/db_1 Type[rac]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /oracle/app/oracle/product/19.3.0/db_1, host: cdjkdb1.
Command failed: /oracle/app/oracle/product/19.3.0/db_1/OPatch/opatchauto apply /oracle/patch/33567274/33509923 -oh /oracle/app/oracle/product/19.3.0/db_1 -target_type rac_database -binary -invPtrLoc /oracle/app/oracle/product/19.3.0/db_1/oraInst.loc -jre /oracle/app/oracle/product/19.3.0/db_1/OPatch/jre -persistresult /oracle/app/oracle/product/19.3.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_cdjkdb1_rac_2.ser -analyzedresult /oracle/app/oracle/product/19.3.0/db_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_cdjkdb1_rac_2.ser
Command failure output:
==Following patches FAILED in apply:
Patch: /oracle/patch/33567274/33509923/33515361
Log: /oracle/app/oracle/product/19.3.0/db_1/cfgtoollogs/opatchauto/core/opatch/opatch2022-10-13_20-49-00PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Thu Oct 13 20:49:54 2022
Time taken to complete the session 3 minutes, 3 seconds
opatchauto failed with error code 42
最初由于忽略了日志的info输出信息,在网上以及mos上找各种方式尝试,均已失败告终,随后重新研究日志信息,定位了原因,最终顺利升级。
错误方式一:
排除oui-patch.xml文件权限,日志里没有这一项,排查无意义;
错误方式二:
排查oneoff目录下的内容,日志里也没有这一项。
正确方式:
根据日志提示信息,有一个关键项:libxxx.so(详见下图)正在使用,不能被opatch使用的信息,使用fuser libxxx.so查看后,会有显示占用的进程号,
kill -9 进程号
结束进程,再执行就顺利完成了。
值得注意的是,其实在补丁升级的过程中报错信息在日志里已经有明显提示了,所以还是先仔细看日志,然后再根据日志提示去针对性的处理。