Opatch报错，OUI-67035，OUI-67073…

bash-3.00$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/dev/db/tech_st/11.2.0
Central Inventory : /u01/app/oraInventory
   from           : /u01/dev/db/tech_st/11.2.0/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_17_15_34/apply2014-05-06_17- 15-34PM_1.log

Applying interim patch '18544852' to OH '/u01/dev/db/tech_st/11.2.0'
Verifying environment and performing prerequisite checks...
[ Error during Oracle Home discovery Phase]. Detail: OPatch failed: ApplySession failed to prepare the system. orac le/cluster/install/InstallException
Log file location: /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_17_15_34/apply2014-05-06_17-1 5-34PM_1.log

Recommended actions: Please make sure no other OPatch or OUI processes is running. Try to run $ORACLE_HOME/oui/bin/ runInstsaller.

OPatch failed with error code 20

日志显示：

May 6, 2014 5:15:41 PM]     Size of directory "/u01/dev/db/tech_st/11.2.0/.patch_storage" after cleanup is 98368496 bytes.
[May 6, 2014 5:15:41 PM]     UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
                             deleted, Please refer log file.
[May 6, 2014 5:15:41 PM]     [ Error during Oracle Home discovery phase ] Detail:                              OUI-67073:OPatch failed: ApplySession failed to prepare the system. oracle/cluster/install/InstallException
[May 6, 2014 5:15:41 PM]     OUI-67035:System is intact, OPatch will not restore the system
[May 6, 2014 5:15:41 PM]     Finishing ApplySession at Tue May 06 17:15:41 CST 2014
[May 6, 2014 5:15:41 PM]     Total time spent waiting for user-input is 0 seconds.  Finish at Tue May 06 17:15:41 CST 2014
[May 6, 2014 5:15:41 PM]     Log file location: /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_17_15_34/apply2014-05-06_17-15-34PM_1.log
[May 6, 2014 5:15:41 PM]     Stack Description: java.lang.RuntimeException: ApplySession failed to prepare the system. oracle/cluster/install/InstallException
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgClusterwareInfo.createClusterwareInfo(OiipgClusterwareInfo.java:155)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgClusterwareInfo.getClusterwareInfo(OiipgClusterwareInfo.java:180)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgDetectCluster.(OiipgDetectCluster.java:131)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgDetectCluster.getDetectCluster(OiipgDetectCluster.java:146)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.opatch.Rac.getLocalNode(Rac.java:1789)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.opatch.Rac.getInstance(Rac.java:1236)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.opatch.ApplySession.processLocal(ApplySession.java:3758)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.opatch.ApplySession.process(ApplySession.java:5939)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2063)
[May 6, 2014 5:15:41 PM]     StackTrace: oracle.opatch.OPatch.main(OPatch.java:613)
bash-3.00$
bash-3.00$

--查找资料，原因如下:

Cause
Opatch is trying to get local and remote node but fails and throws exception.

The inventory.xml file (in /ContentsXML) has an inconsistent nodelist.

 

--发现节点信息不对，更新inventory.xml节点信息

bash-3.00$ runInstaller -updateNodeList -local -silent CLUSTER_NODES=uat1,uat2 ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 LOCAL_NODE=uat1
You do not have sufficient permissions to access the inventory '/u01/app/oraInventory/oraInstaller.properties'. Installation cannot continue. It is required that the primary group of the install user is same as the inventory owner group. Make sure that the install user is part of the inventory owner group and restart the installer.: Permission denied
bash-3.00$ runInstaller -updateNodeList -local -silent CLUSTER_NODES=uat1,uat2 ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 LOCAL_NODE=uat1
You do not have sufficient permissions to access the inventory '/u01/app/oraInventory/Contents'. Installation cannot continue. It is required that the primary group of the install user is same as the inventory owner group. Make sure that the install user is part of the inventory owner group and restart the installer.: Permission denied

 

--权限不对，修改 inventory.xml文件权限，继续

 

bash-3.00$ runInstaller -updateNodeList -local -silent CLUSTER_NODES=uat1,uat2 ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 LOCAL_NODE=uat1
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 171287 MB    Passed
The inventory pointer is located at /var/opt/oracle/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
bash-3.00$

 

--第二个节点执行

$ORACLE_HOME/oui/bin/runInstaller -updateNodeList -local -silent CLUSTER_NODES=uat1,uat2 ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 LOCAL_NODE=uat2

--继续

ash-3.00$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/dev/db/tech_st/11.2.0
Central Inventory : /u01/app/oraInventory
   from           : /u01/dev/db/tech_st/11.2.0/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_18_23_29/apply2014-05-06_18-23-29PM_1.log

Applying interim patch '18544852' to OH '/u01/dev/db/tech_st/11.2.0'
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:

Following executables are active :
/u01/dev/db/tech_st/11.2.0/lib/libclntsh.so.11.1
Log file location: /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_18_23_29/apply2014-05-06_18-23-29PM_1.log

Recommended actions: OPatch needs to modify files which are being used by some processes.

OPatch failed with error code 41

 

pplying interim patch '18544852' to OH '/u01/dev/db/tech_st/11.2.0'
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:

Following executables are active :
/u01/dev/db/tech_st/11.2.0/lib/libclntsh.so.11.1
Log file location: /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_18_34_15/apply2014-05-06_18-34-14PM_1.log

Recommended actions: OPatch needs to modify files which are being used by some processes.

OPatch failed with error code 41

 

--查找进程，OPatch needs to modify files which are being used by some processes.

bash-3.00$ ps -fu oradev
     UID   PID  PPID   C    STIME TTY         TIME CMD
  oradev  2704  2703   0   Apr 28 pts/19      0:11 rman target /
  oradev  8836  5777   0   Mar 13 pts/13      0:00 -sh
  oradev  6799  6795   0   Apr 29 pts/16      0:00 bash
  oradev  8115  9661   0   Mar 14 pts/9       0:00 -sh
  oradev  8840  8836   0   Mar 13 pts/13      0:01 bash
  oradev 28795 28794   0   Apr 24 pts/18      0:00 -sh
  oradev  8119  8115   0   Mar 14 pts/9       0:01 bash
  oradev 29314 15839   0   Apr 29 pts/12      0:00 -sh
  oradev 29320 29314   0   Apr 29 pts/12      0:00 bash
  oradev  2699  2697   0   Apr 28 pts/19      0:00 -sh
  oradev  6795 25212   0   Apr 29 pts/16      0:00 -sh
  oradev 16504  8840   0 18:35:29 pts/13      0:00 ps -fu oradev
  oradev 28799 28795   0   Apr 24 pts/18      0:00 bash
  oradev  2703  2699   0   Apr 28 pts/19      0:00 bash

 

 -- Kill rman连接进程，继续

 

bash-3.00$ kill 2704
bash-3.00$ ps -fu oradev
     UID   PID  PPID   C    STIME TTY         TIME CMD
  oradev 16506  8840   0 18:35:52 pts/13      0:00 ps -fu oradev
  oradev  8836  5777   0   Mar 13 pts/13      0:00 -sh
  oradev  6799  6795   0   Apr 29 pts/16      0:00 bash
  oradev  8115  9661   0   Mar 14 pts/9       0:00 -sh
  oradev  8840  8836   0   Mar 13 pts/13      0:01 bash
  oradev 28795 28794   0   Apr 24 pts/18      0:00 -sh
  oradev  8119  8115   0   Mar 14 pts/9       0:01 bash
  oradev 29314 15839   0   Apr 29 pts/12      0:00 -sh
  oradev 29320 29314   0   Apr 29 pts/12      0:00 bash
  oradev  2699  2697   0   Apr 28 pts/19      0:00 -sh
  oradev  6795 25212   0   Apr 29 pts/16      0:00 -sh
  oradev 28799 28795   0   Apr 24 pts/18      0:00 bash
  oradev  2703  2699   0   Apr 28 pts/19      0:00 bash
  oradev  2703  2699   0   Apr 28 pts/19      0:00 bash
bash-3.00$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/dev/db/tech_st/11.2.0
Central Inventory : /u01/app/oraInventory
   from           : /u01/dev/db/tech_st/11.2.0/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_18_36_09/apply2014-05-06_18-36-09PM_1.log

Applying interim patch '18544852' to OH '/u01/dev/db/tech_st/11.2.0'
Verifying environment and performing prerequisite checks...
All checks passed.

This node is part of an Oracle Real Application Cluster.
Remote nodes: 'uat2'
Local node: 'uat1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/dev/db/tech_st/11.2.0')

Is the local system ready for patching? [y|n]

Applying interim patch '18544852' to OH '/u01/dev/db/tech_st/11.2.0'
Verifying environment and performing prerequisite checks...
All checks passed.

This node is part of an Oracle Real Application Cluster.
Remote nodes: 'uat2'
Local node: 'uat1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/dev/db/tech_st/11.2.0')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...

Patching component oracle.rdbms.rsf, 11.2.0.3.0...

Patching component oracle.rdbms, 11.2.0.3.0...

OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ld: warning: symbol `_start' has differing types:
        (file /u01/dev/db/tech_st/11.2.0/lib/prod/lib/v9/crt1.o type=FUNC; file /u01/dev/db/tech_st/11.2.0/lib//libserver11.a(skds.o) type=OBJT);

 

Verifying the update...

The local system has been patched.  You can restart Oracle instances on it.

Patching in rolling mode.

The node 'uat2' will be patched next.

Please shutdown Oracle instances running out of this ORACLE_HOME on 'uat2'.
(Oracle Home = '/u01/dev/db/tech_st/11.2.0')

Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'uat2'
   Apply-related files are:
     FP = "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_files.txt"
     DP = "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_dirs.txt"
     MP = "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/make_cmds.txt"
     RC = "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/remote_cmds.txt"

Instantiating the file "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...

OPatch failed to copy files to remote nodes 'uat2' .  Detail: Error while copying files inside directory '/u01/dev/db/tech_st/11.2.0' based on listed file '/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_files.txt.instantiated' to nodes 'uat2'. [PRKC-1073 : Failed to transfer directory "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_files.txt.instantiated" to any of the given nodes "uat2 ".
Error on node uat2:PRKC-1044 : Failed to check remote command execution setup for node uat2 using shells /usr/bin/ssh and /usr/bin/rsh
permission denied]

OPatch failed to copy files to remote nodes 'uat2' .
Do you want to proceed? [y|n]
y
User Responded with: Y
Instantiating the file "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/dev/db/tech_st/11.2.0/.patch_storage/18544852_May_5_2014_15_19_11/rac/make_cmds.txt" with actual path.
Running command on remote node 'uat2':
cd /u01/dev/db/tech_st/11.2.0/network/lib; /usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'uat2':
cd /u01/dev/db/tech_st/11.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'uat2':
cd /u01/dev/db/tech_st/11.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk irenamedg ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'uat2':
cd /u01/dev/db/tech_st/11.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/dev/db/tech_st/11.2.0 || echo REMOTE_MAKE_FAILED::>&2

The node 'uat2' has been patched.  You can restart Oracle instances on it.

There were relinks on remote nodes.  Remember to check the binary size and timestamp on the nodes 'uat2' .
The following make commands were invoked on remote nodes:
'cd /u01/dev/db/tech_st/11.2.0/network/lib; /usr/ccs/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/dev/db/tech_st/11.2.0
cd /u01/dev/db/tech_st/11.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/dev/db/tech_st/11.2.0
cd /u01/dev/db/tech_st/11.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk irenamedg ORACLE_HOME=/u01/dev/db/tech_st/11.2.0
cd /u01/dev/db/tech_st/11.2.0/rdbms/lib; /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/dev/db/tech_st/11.2.0
'

Patch 18544852 successfully applied
OPatch Session completed with warnings.
Log file location: /u01/dev/db/tech_st/11.2.0/cfgtoollogs/opatch/18544852_May_06_2014_18_36_09/apply2014-05-06_18-36-09PM_1.log

OPatch completed with warnings.

--- OK，总算打上了，Patch 18544852 successfully applied

参考文档：

patch Apply Fails with Error Code 20 during Oracle Home Discovery Phase (文档 ID 1631715.1) 转到底部 

--------------------------------------------------------------------------------

修改时间:2014-3-25类型:PROBLEM    
 

In this Document

 Symptoms
 Changes
 Cause
 Solution
 References

--------------------------------------------------------------------------------

 

Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

Symptoms
 Opatch online patching fails with error code 20 during Oracle Home discovery Phase:

 

opatch apply online -connectString RACDB1:sys:test:ORACLERAC1,RACDB2:sys:test:ORACLERAC2

Oracle Interim Patch Installer version 11.2.0.3.4

Copyright (c) 2012, Oracle Corporation.  All rights reserved.

 

The patch should be applied/rolled back in '-all_nodes' mode only.

Converting the RAC mode to '-all_nodes' mode.

Applying interim patch '13984324' to OH '/u01/app/oracle/product/11.2.0.3/dbhome_7'

Verifying environment and performing prerequisite checks...

[ Error during Oracle Home discovery Phase]. Detail: OPatch failed: ApplySession failed to prepare the system. oracle/cluster/install/InstallException

Log file location: /u01/app/oracle/product/11.2.0.3/dbhome_7/cfgtoollogs/opatch/13984324_Feb_03_2014_10_20_26/apply2014-02-03_10-20-26AM_1.log

 

Recommended actions: Please make sure no other OPatch or OUI processes is running. Try to run $ORACLE_HOME/oui/bin/runInstaller.

OPatch failed with error code 20

 

In Apply log : 

[Jan 29, 2014 2:52:01 PM] [ Error during Oracle Home discovery phase ] Detail: OUI-67073:OPatch failed: ApplySession failed to prepare the system. oracle/cluster/install/InstallException
[Jan 29, 2014 2:52:01 PM] OUI-67035:System is intact, OPatch will not restore the system
[Jan 29, 2014 2:52:01 PM] Finishing ApplySession at Wed Jan 29 14:52:01 GMT 2014
[Jan 29, 2014 2:52:01 PM] Total time spent waiting for user-input is 0 seconds. Finish at Wed Jan 29 14:52:01 GMT 2014
[Jan 29, 2014 2:52:01 PM] Log file location: /u01/app/oracle/product/11.2.0.3/dbhome_7/cfgtoollogs/opatch/13984324_Jan_29_2014_14_51_55/apply2014-01-29_14-51-55PM_1.log
[Jan 29, 2014 2:52:01 PM] Stack Description: java.lang.RuntimeException: ApplySession failed to prepare the system. oracle/cluster/install/InstallException
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgClusterwareInfo.createClusterwareInfo(OiipgClusterwareInfo.java:155)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgClusterwareInfo.getClusterwareInfo(OiipgClusterwareInfo.java:180)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgDetectCluster.(OiipgDetectCluster.java:131)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.sysman.oii.oiip.oiipg.OiipgDetectCluster.getDetectCluster(OiipgDetectCluster.java:146)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.opatch.Rac.getLocalNode(Rac.java:1789)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.opatch.Rac.getInstance(Rac.java:1236)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.opatch.ApplySession.processLocal(ApplySession.java:3758)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.opatch.ApplySession.process(ApplySession.java:5939)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2063)
[Jan 29, 2014 2:52:01 PM] StackTrace: oracle.opatch.OPatch.main(OPatch.java:613)

 

Changes
Applying an interim patch on RAC environment.

Cause
Opatch is trying to get local and remote node but fails and throws exception.

The inventory.xml file (in /ContentsXML) has an inconsistent nodelist.

Solution
Before patch application, the nodelist must be updated to reflect the correct node names on all RAC nodes.
 

On a 2 node RAC, update the inventory as follows: 
 

On Node 1 :

  $ORACLE_HOME/oui/bin/runInstaller -updateNodeList -local -silent CLUSTER_NODES=dbnode1,dbnode2 ORACLE_HOME=$ORACLE_HOME LOCAL_NODE=dbnode1 
 

On Node 2 :
 
  $ORACLE_HOME/oui/bin/runInstaller -updateNodeList -local -silent CLUSTER_NODES=dbnode1,dbnode2 ORACLE_HOME=$ORACLE_HOME LOCAL_NODE=dbnode2