Prerequisite check "CheckActiveFilesAndExecutables" failed Following active executables are not used

oracle19.3.0.0 rac+centos7.6

补丁为19.5.0.0

报错现象：

节点2打补丁失败。

[grid@rac2 ~]$ cd /oracle/app/patch/30116789/30125133
[grid@rac2 30125133]$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2019, Oracle Corporation.  All rights reserved.

Oracle Home       : /oracle/app/19c/grid
Central Inventory : /oracle/app/oraInventory
   from           : /oracle/app/19c/grid/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.2.0.7.0
Log file location : /oracle/app/19c/grid/cfgtoollogs/opatch/opatch2019-11-15_12-15-38PM_1.log

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:

Following active executables are not used by opatch process :
/oracle/app/19c/grid/lib/libclntsh.so.19.1
/oracle/app/19c/grid/lib/libasmclntsh19.so

Following active executables are used by opatch process :

/oracle/app/19c/grid/bin/tnslsnr
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Log file location: /oracle/app/19c/grid/cfgtoollogs/opatch/opatch2019-11-15_12-15-38PM_1.log

OPatch failed with error code 73

 

处理办法：

停掉数据库和监听，手动kill掉所有grid和oracle进程，再打成功！

[root@rac2 ~]# /oracle/app/19c/grid/bin/crsctl stop crs -f
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
[root@rac2 ~]# ps -ef|grep d.bin
grid      18199      1  0 12:11 ?        00:00:03 /oracle/app/19c/grid/bin/evmd.bin
grid      18200      1  0 12:11 ?        00:00:01 /oracle/app/19c/grid/bin/mdnsd.bin
grid      18341      1  0 12:11 ?        00:00:01 /oracle/app/19c/grid/bin/gpnpd.bin
grid      19128      1  0 12:11 ?        00:00:03 /oracle/app/19c/grid/bin/gipcd.bin
root      19664      1  0 12:12 ?        00:00:01 /oracle/app/19c/grid/bin/cssdmonitor
root      19666      1  0 12:12 ?        00:00:04 /oracle/app/19c/grid/bin/osysmond.bin
root      19698      1  0 12:12 ?        00:00:01 /oracle/app/19c/grid/bin/cssdagent
grid      19715      1  1 12:12 ?        00:00:05 /oracle/app/19c/grid/bin/ocssd.bin
root      19977      1  0 12:12 ?        00:00:03 /oracle/app/19c/grid/bin/octssd.bin reboot
root      20030      1  1 12:12 ?        00:00:06 /oracle/app/19c/grid/bin/crsd.bin reboot
root      20110      1  0 12:12 ?        00:00:03 /oracle/app/19c/grid/bin/orarootagent.bin
grid      20218      1  0 12:12 ?        00:00:03 /oracle/app/19c/grid/bin/oraagent.bin
grid      20291      1  0 12:12 ?        00:00:00 /oracle/app/19c/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid      21551  18199  0 12:13 ?        00:00:01 /oracle/app/19c/grid/bin/evmlogger.bin -o /oracle/app/19c/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /oracle/app/19c/grid/log/[HOSTNAME]/evmd/evmlogger.log
root      24012   9790  0 12:20 pts/0    00:00:00 grep --color=auto d.bin
[root@rac2 ~]# ps -ef|grep d.bin|awk '{print $2}'
18199
18200
18341
19128
19664
19666
19698
19715
19977
20030
20110
20218
20291
21551
24082
[root@rac2 ~]# kill -9 18199
[root@rac2 ~]# kill -9 18200
[root@rac2 ~]# kill -9 18341
[root@rac2 ~]# kill -9 19128
[root@rac2 ~]# kill -9 19664
[root@rac2 ~]# kill -9 19666
[root@rac2 ~]# kill -9 19698
[root@rac2 ~]# kill -9 19715
[root@rac2 ~]# kill -9 19977
[root@rac2 ~]# kill -9 20030
[root@rac2 ~]# kill -9 20110
-bash: kill: (20110) - No such process
[root@rac2 ~]# kill -9 20218
-bash: kill: (20218) - No such process
[root@rac2 ~]# kill -9 20291
[root@rac2 ~]# kill -9 21551
-bash: kill: (21551) - No such process
[root@rac2 ~]# kill -9 24082
-bash: kill: (24082) - No such process
[root@rac2 ~]# 
[root@rac2 ~]# 
[root@rac2 ~]# ps -ef|grep d.bin
root      24232   9790  0 12:20 pts/0    00:00:00 grep --color=auto d.bin
[root@rac2 ~]# ps -ef|grep oracle
grid      20254      1  1 12:12 ?        00:00:09 /oracle/app/19c/grid/jdk/bin/java -server -Xms30M -Xmx512M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -DTRACING.ENABLED=false -XX:ParallelGCThreads=1 -cp /oracle/app/19c/grid/jlib/cha.jar:/oracle/app/19c/grid/jlib/chaconfig.jar:/oracle/app/19c/grid/jlib/cha-diag-msg.jar:/oracle/app/19c/grid/jlib/clsce.jar:/oracle/app/19c/grid/jlib/srvm.jar:/oracle/app/19c/grid/jlib/srvmhas.jar:/oracle/app/19c/grid/jlib/srvmasm.jar:/oracle/app/19c/grid/jlib/netcfg.jar:/oracle/app/19c/grid/jdbc/lib/ojdbc8.jar:/oracle/app/19c/grid/ucp/lib/ucp.jar:/oracle/app/19c/grid/jlib/fte.jar:/oracle/app/19c/grid/jlib/jai_core.jar:/oracle/app/19c/grid/jlib/mlibwrapper_jai.jar:/oracle/app/19c/grid/jlib/vecmath.jar:/oracle/app/19c/grid/jlib/jai_codec.jar:/oracle/app/19c/grid/jlib/jh.jar oracle.cha.server.CHADDriver
grid      20275      1  0 12:12 ?        00:00:00 /oracle/app/19c/grid/opmn/bin/ons -d
grid      20393  20275  0 12:12 ?        00:00:00 /oracle/app/19c/grid/opmn/bin/ons -d
root      24249   9790  0 12:20 pts/0    00:00:00 grep --color=auto oracle
[root@rac2 ~]# kill -9 20254
[root@rac2 ~]# kill -9 20275
[root@rac2 ~]# kill -9 20393
[root@rac2 ~]# ps -ef|grep grid
root      22113  10401  0 12:14 pts/1    00:00:00 su - grid
grid      22114  22113  0 12:14 pts/1    00:00:00 -bash
root      24343   9790  0 12:21 pts/0    00:00:00 grep --color=auto grid
[root@rac2 ~]# ps -ef|grep oracle
root      24360   9790  0 12:21 pts/0    00:00:00 grep --color=auto oracle
[root@rac2 ~]# 

成功结果：

[grid@rac2 30125133]$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2019, Oracle Corporation.  All rights reserved.

Oracle Home       : /oracle/app/19c/grid
Central Inventory : /oracle/app/oraInventory
   from           : /oracle/app/19c/grid/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.2.0.7.0
Log file location : /oracle/app/19c/grid/cfgtoollogs/opatch/opatch2019-11-15_12-16-08PM_1.log

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:

Following active executables are not used by opatch process :
/oracle/app/19c/grid/lib/libclntsh.so.19.1
/oracle/app/19c/grid/lib/libasmclntsh19.so

Following active executables are used by opatch process :

/oracle/app/19c/grid/bin/tnslsnr
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Log file location: /oracle/app/19c/grid/cfgtoollogs/opatch/opatch2019-11-15_12-16-08PM_1.log

OPatch failed with error code 73
[grid@rac2 30125133]$ ls -l /oracle/app/19c/grid/lib/libclntsh.so.19.1
-rw-r--r-- 1 grid oinstall 79927312 Nov 14 12:36 /oracle/app/19c/grid/lib/libclntsh.so.19.1
[grid@rac2 30125133]$ ps -ef|grep pmon
grid      23601  22114  0 12:18 pts/1    00:00:00 grep --color=auto pmon
[grid@rac2 30125133]$ ps -ef|grep sqlplus
grid      23623  22114  0 12:18 pts/1    00:00:00 grep --color=auto sqlplus
[grid@rac2 30125133]$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2019, Oracle Corporation.  All rights reserved.

Oracle Home       : /oracle/app/19c/grid
Central Inventory : /oracle/app/oraInventory
   from           : /oracle/app/19c/grid/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.2.0.7.0
Log file location : /oracle/app/19c/grid/cfgtoollogs/opatch/opatch2019-11-15_12-21-33PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30125133  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/oracle/app/19c/grid')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30125133' to OH '/oracle/app/19c/grid'
ApplySession: Optional component(s) [ oracle.ctx, 19.0.0.0.0 ] , [ oracle.network.gsm, 19.0.0.0.0 ] , [ oracle.rdbms.ic, 19.0.0.0.0 ] , [ oracle.sdo, 19.0.0.0.0 ] , [ oracle.network.cman, 19.0.0.0.0 ] , [ oracle.ctx.atg, 19.0.0.0.0 ] , [ oracle.rdbms.oci, 19.0.0.0.0 ] , [ oracle.net.cman, 19.0.0.0.0 ] , [ oracle.options.olap.awm, 19.0.0.0.0 ] , [ oracle.xdk, 19.0.0.0.0 ] , [ oracle.oraolap.mgmt, 19.0.0.0.0 ] , [ oracle.dbdev, 19.0.0.0.0 ] , [ oracle.rdbms.scheduler, 19.0.0.0.0 ] , [ oracle.rdbms.dv, 19.0.0.0.0 ] , [ oracle.assistants.usm, 19.0.0.0.0 ] , [ oracle.rdbms.install.seeddb, 19.0.0.0.0 ] , [ oracle.odbc, 19.0.0.0.0 ] , [ oracle.marvel, 19.0.0.0.0 ] , [ oracle.precomp.common, 19.0.0.0.0 ] , [ oracle.precomp.lang, 19.0.0.0.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 19.0.0.0.0...

Patching component oracle.rdbms.rsf, 19.0.0.0.0...

Patching component oracle.assistants.acf, 19.0.0.0.0...

Patching component oracle.assistants.deconfig, 19.0.0.0.0...

Patching component oracle.assistants.server, 19.0.0.0.0...

Patching component oracle.buildtools.rsf, 19.0.0.0.0...

Patching component oracle.ldap.rsf, 19.0.0.0.0...

Patching component oracle.network.rsf, 19.0.0.0.0...

Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...

Patching component oracle.sqlplus, 19.0.0.0.0...

Patching component oracle.ldap.rsf.ic, 19.0.0.0.0...

Patching component oracle.rdbms.rman, 19.0.0.0.0...

Patching component oracle.rdbms.util, 19.0.0.0.0...

Patching component oracle.ovm, 19.0.0.0.0...

Patching component oracle.network.listener, 19.0.0.0.0...

Patching component oracle.rdbms.install.plugins, 19.0.0.0.0...

Patching component oracle.dbjava.jdbc, 19.0.0.0.0...

Patching component oracle.rdbms.deconfig, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf, 19.0.0.0.0...

Patching component oracle.oraolap.dbscripts, 19.0.0.0.0...

Patching component oracle.install.deinstalltool, 19.0.0.0.0...

Patching component oracle.dbjava.ic, 19.0.0.0.0...

Patching component oracle.sdo.locator, 19.0.0.0.0...

Patching component oracle.ons, 19.0.0.0.0...

Patching component oracle.ldap.security.osdt, 19.0.0.0.0...

Patching component oracle.ctx.rsf, 19.0.0.0.0...

Patching component oracle.duma, 19.0.0.0.0...

Patching component oracle.ldap.owm, 19.0.0.0.0...

Patching component oracle.oracore.rsf, 19.0.0.0.0...

Patching component oracle.assistants.asm, 19.0.0.0.0...

Patching component oracle.sdo.locator.jrf, 19.0.0.0.0...

Patching component oracle.network.client, 19.0.0.0.0...

Patching component oracle.tfa, 19.0.0.0.0...

Patching component oracle.sqlplus.ic, 19.0.0.0.0...

Patching component oracle.dbjava.ucp, 19.0.0.0.0...

Patching component oracle.xdk.rsf, 19.0.0.0.0...

Patching component oracle.xdk.parser.java, 19.0.0.0.0...

Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf.core, 19.0.0.0.0...
Patch 30125133 successfully applied.
Sub-set patch [29517242] has become inactive due to the application of a super-set patch [30125133].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /oracle/app/19c/grid/cfgtoollogs/opatch/opatch2019-11-15_12-21-33PM_1.log

OPatch succeeded.
[grid@rac2 30125133]$ opatch lspatches
30125133;Database Release Update : 19.5.0.0.191015 (30125133)
30122167;ACFS RELEASE UPDATE 19.5.0.0.0 (30122167)
30122149;OCW RELEASE UPDATE 19.5.0.0.0 (30122149)
29401763;TOMCAT RELEASE UPDATE 19.0.0.0.0 (29401763)

OPatch succeeded.

 

注意使用fuser xxx不好使。

官方参考不太好使：

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

SYMPTOMS

opatch auto failed with error:

Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/u01/app/11.2.0/grid/lib/libclntsh.so.11.1

OPatch failed with error code 73
 

CAUSE

This library is still being referenced by an active process:
/u01/app/11.2.0/grid/lib/libclntsh.so.11.1
 
This error during apply is indicative of the cause of the problem:

Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/u01/app/11.2.0/grid/lib/libclntsh.so.11.1
 

SOLUTION

As per opatch*.log, there are still processes using one of the files that is being patched:
Following executables are active :
/u01/app/11.2.0/grid/lib/libclntsh.so.11.1

ACTION PLAN
~~~~~~~~~~~
1. As 'root' user, identify process using this file/library:
  # fuser /u01/app/11.2.0/grid/lib/libclntsh.so.11.1
  # lsof /u01/app/11.2.0/grid/lib/libclntsh.so.11.1

2. You may shutdown application or kill process using above file/library before trying opatch again.
  # crsctl stop crs -f
  # fuser /u01/app/11.2.0/grid/lib/libclntsh.so.11.1
  # kill -9 <ospid of process identified above>