ORACLE opatch 打补丁fuser command output for /u01/....../bin/crsctl.bin is Failure

    最近，本地测试数据库oracle 11.2.0.4 for rhel7打补丁的时候失败了，检查opatch应用补丁的日志显示错误提示如下：

CRS-4133: Oracle High Availability Services has been stopped.

Successfully unlock /u01/app/11.2.0/grid

 2020-06-26 17:05:46: Invoking removeproc to clean oracle client procs

 2020-06-26 17:05:46: Executing cmd: /sbin/fuser -k /u01/app/11.2.0/grid/bin/crsctl.bin

 2020-06-26 17:05:46: Command output:

>  Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k /u01/app/11.2.0/grid/bin/crsctl.bin

>End Command output

 2020-06-26 17:05:46: fuser command output for /u01/app/11.2.0/grid/bin/crsctl.bin is Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k

/u01/app/11.2.0/grid/bin/crsctl.bin

 2020-06-26 17:05:46: Oracle user for /u01/app/11.2.0/grid is grid

 2020-06-26 17:05:46: Opening file /etc/oracle/ocr.loc

 2020-06-26 17:05:46: Value (FALSE) is set for key=local_only

 2020-06-26 17:05:46: Running as user grid: /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26925576 -oh /u01/app/11.2.0/grid

 2020-06-26 17:05:46: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26925576 -oh /u01/app/11.2.0

/grid '

 2020-06-26 17:05:47: Removing file /tmp/filewsfQVx

 2020-06-26 17:05:47: Successfully removed file: /tmp/filewsfQVx

 2020-06-26 17:05:47: /bin/su exited with rc=2

 2020-06-26 17:05:47: Error : The opatch Applicable check failed.  The patch /tmp/psu/27107360/26925576 is not applicable to /u01/app/11.2.0/grid

 2020-06-26 17:05:47: Running as user grid: /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26609929 -oh /u01/app/11.2.0/grid

 2020-06-26 17:05:47: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26609929 -oh /u01/app/11.2.0

/grid '

 2020-06-26 17:05:48: Removing file /tmp/filePKM5qO

 2020-06-26 17:05:48: Successfully removed file: /tmp/filePKM5qO

 2020-06-26 17:05:48: /bin/su exited with rc=2

 

 2020-06-26 17:05:48: Error : The opatch Applicable check failed.  The patch /tmp/psu/27107360/26609929 is not applicable to /u01/app/11.2.0/grid

 2020-06-26 17:05:48: Running as user grid: /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/22502505 -oh /u01/app/11.2.0/grid

 2020-06-26 17:05:48: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/22502505 -oh /u01/app/11.2.0

/grid '

 2020-06-26 17:05:49: Removing file /tmp/filecn51l7

 2020-06-26 17:05:49: Successfully removed file: /tmp/filecn51l7

 2020-06-26 17:05:49: /bin/su exited with rc=2

 2020-06-26 17:05:49: Error : The opatch Applicable check failed.  The patch /tmp/psu/27107360/22502505 is not applicable to /u01/app/11.2.0/grid

 2020-06-26 17:05:49: Status of Applicable  check  for /u01/app/11.2.0/grid is 0

 2020-06-26 17:05:49: Error:Patch Applicable check failed for /u01/app/11.2.0/grid

 2020-06-26 17:05:49: Executing cmd: /bin/rpm -q sles-release

 2020-06-26 17:05:49: Command output:

>  package sles-release is not installed

 

    观察日志，提示比较明显的错误信息是：The opatch Applicable check failed.  The patch /tmp/psu/27107360/...... is not applicable to /u01/app/11.2.0/grid。

按照以往的补丁应用经验，按照如下过程进行异常排查：

    1、重新上传补丁包，排除补丁包不完整的情况

    2、赋予补丁集777权限，grid解压，排除补丁包宿主权限问题

    3、将补丁集放在单独的空目录，grid执行解压

    经过上诉三个步骤，在应用补丁集时还是提示相同的错误，仔细观察补丁集应用日志，发现如下观察错误提示：

 2020-06-26 17:05:46: Command output:

>  Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k /u01/app/11.2.0/grid/bin/crsctl.bin

>End Command output

 2020-06-26 17:05:46: fuser command output for /u01/app/11.2.0/grid/bin/crsctl.bin is Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k

/u01/app/11.2.0/grid/bin/crsctl.bin

    难道是/sbin/fuser命令不存在，经过排查，发现/sbin/fuser命令真的不在，死马当活马医，执行yum install psmisc，在集群

两个节点安装了psmisc.x86_64 0:22.20-15.el7，再次执行补丁集应用，补丁集应用成功。

# /u01/app/11.2.0/grid/OPatch/opatch auto /u01/soft/27107360 -oh /u01/app/11.2.0/grid

Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /u01/soft -patchn 27107360 -oh /u01/app/11.2.0/grid -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto 2020-06-26_18-20-05.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:

/u01/app/11.2.0/grid/cfgtoollogs/opatchauto 2020-06-26_18-20-05.report.log

 2020-06-26 18:20:05: Starting Clusterware Patch Setup

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Stopping CRS...

Stopped CRS successfully

patch /u01/soft/27107360/26925576  apply successful for home  /u01/app/11.2.0/grid

patch /u01/soft/27107360/26609929  apply successful for home  /u01/app/11.2.0/grid

patch /u01/soft/27107360/22502505  apply successful for home  /u01/app/11.2.0/grid

Starting CRS...

Installing Trace File Analyzer

ACFS driver install actions failed

CRS-4123: Oracle High Availability Services has been started.

opatch auto succeeded.