java psu_PSU实施异常解决

--RAC实施

1：问题

[root@andy_node1 OPatch]# ./opatch auto /data1/media/20130820/psu07 -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp

patch /data1/media/20130820/psu07/16619898/custom/server/16619898  apply successful for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1

patch /data1/media/20130820/psu07/16619892  apply failed  for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1

patch /data1/media/20130820/psu07/16619898  apply successful for home  /data1/oracle/app/11.2.0/grid

patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/oracle/app/11.2.0/grid

2：查看详细日志

[root@andy_node1 OPatch]# vi /data1/oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2013-08-25_10-41-06.log

: Opening file /etc/oracle/ocr.loc

2013-08-25 10:42:39: Value (FALSE) is set for key=local_only

oralce:

$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619898/custom/server/16619898 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1

$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1

$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619898/custom/server/16619898 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1

$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1

grid:

$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619898 -oh /data1/oracle/app/11.2.0/grid

$opatch prereq CheckComponents -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/11.2.0/grid

$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619898 -oh /data1/oracle/app/11.2.0/grid

$opatch prereq CheckConflictAgainstOH -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/11.2.0/grid

oracle:

$opatch prereq CheckApplicable -ph /data1/media/20130820/psu07/16619898/custom/server/16619898 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1

$opatch prereq CheckApplicable -ph /data1/media/20130820/psu07/16619892 -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1

srvctl stop home -o /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -s /data1/oracle/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n andy_node1

/sbin/fuser -k /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/oracle

/bin/su oracle -c ' /data1/media/20130820/psu07/16619898/custom/server/16619898/custom/scripts/prepatch.sh -dbhome /data1/oracle/app/oracle/product/11.2.0/dbhome_1 '

/bin/su oracle -c ' /data1/oracle/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /data1/media/20130820/psu07/16619898/custom/server/16619898

-local -silent -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /data1/oracle/app/oracle/product/11.2.0/dbhome_1/oraInst.loc '

/bin/su oracle -c ' /data1/oracle/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /data1/media/20130820/psu07/16619892

-local -silent -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /data1/oracle/app/oracle/product/11.2.0/dbhome_1/oraInst.loc '

/bin/su exited with rc=73

Verifying environment and performing prerequisite checks...

Prerequisite check "CheckActiveFilesAndExecutables" failed.

The details are:

Following executables are active :

/data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1

UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Log file location: /data1/oracle/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-08-25_10-43-47AM_1.log

OPatch failed with error code 73

2013-08-25 10:43:53: patch /data1/media/20130820/psu07/16619892  apply failed  for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1

[root@andy_node1 ~]# vi /data1/oracle/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-08-25_10-43-47AM_1.log

OPatch invoked as follows: 'napply /data1/media/20130820/psu07/16619892

-local -silent -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp -oh /data1/oracle/app/oracle/product/11.2.0/dbhome_1 -invPtrLoc /data1/oracle/app/oracle/product/11.2.0/dbhome_1/oraInst.loc '

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/oracle at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/oracle at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/renamedg at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/renamedg at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/nid at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/nid at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/proc at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/proc at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/rman at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/rman at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/wrap at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/wrap at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/plshprof at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/plshprof at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 at Sun Aug 25 10:43:53 CST 2013

Following executables are active :

/data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1

Prerequisite check "CheckActiveFilesAndExecutables" failed.

The details are:

Following executables are active :

/data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/amdu at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/amdu at Sun Aug 25 10:43:53 CST 2013

Start fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/kfed at Sun Aug 25 10:43:53 CST 2013

Finish fuser command /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/kfed at Sun Aug 25 10:43:53 CST 2013

OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Finishing UtilSession at Sun Aug 25 10:43:53 CST 2013

Log file location: /data1/oracle/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-08-25_10-43-47AM_1.log

Stack Description: java.lang.RuntimeException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

StackTrace: oracle.opatch.OPatchSessionHelper.runApplyPrereqs(OPatchSessionHelper.java:6420)

StackTrace: oracle.opatch.opatchutil.NApply.process(NApply.java:1587)

StackTrace: oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1136)

StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

StackTrace: sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

StackTrace: java.lang.reflect.Method.invoke(Method.java:592)

StackTrace: oracle.opatch.UtilSession.process(UtilSession.java:322)

StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2346)

StackTrace: oracle.opatch.OPatch.main(OPatch.java:613)

3：日志里就有解决办法

[oracle@andy_node1 lib]$ ls -l libclntsh.so

lrwxrwxrwx 1 oracle oinstall 17 May  7  2012 libclntsh.so -> libclntsh.so.11.1

[root@andy_node1 ~]# /sbin/fuser /data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1

/data1/oracle/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1: 26679m

[root@andy_node1 ~]# ps -ef | grep 26679

root     12370 11255  0 20:36 pts/0    00:00:00 grep 26679

patrol   26679  8362  0 12:17 ?        00:00:03 /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/sqlplus -s

oracle   26682 26679  0 12:17 ?        00:02:08 oracleandydb1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

[root@andy_node1 ~]# ps -ef | grep sqlplus

root     12396 11255  0 20:36 pts/0    00:00:00 grep sqlplus

patrol   26679  8362  0 12:17 ?        00:00:03 /data1/oracle/app/oracle/product/11.2.0/dbhome_1/bin/sqlplus -s

[root@andy_node1 ~]#

4：不停的去kill -9 sqlplus进程，再次执行成功

[root@andy_node1 OPatch]# ./opatch auto /data1/media/20130820/psu07 -ocmrf /data1/oracle/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp

patch /data1/media/20130820/psu07/16619898/custom/server/16619898  apply successful for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1

patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/oracle/app/oracle/product/11.2.0/dbhome_1

patch /data1/media/20130820/psu07/16619898  apply successful for home  /data1/oracle/app/11.2.0/grid

patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/oracle/app/11.2.0/grid

--单机physical dataguard实施

1：dataguard是rman恢复来的，没有加到crs里，实际上加进去，自动open也是open read only with apply模式的。

考虑打rac时crs会自动起停，本次想停掉has和db来打，并想看看能否遇到GI和DB分开打的情况。

[grid@andy_phydg ~]$ crsctl status res -t

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA_DG.dg

ONLINE  ONLINE       andy_phydg

ora.LISTENER.lsnr

ONLINE  ONLINE       andy_phydg

ora.OCR_DG.dg

ONLINE  ONLINE       andy_phydg

ora.asm

ONLINE  ONLINE       andy_phydg              Started

ora.ons

OFFLINE OFFLINE      andy_phydg

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

1        ONLINE  ONLINE       andy_phydg

ora.diskmon

1        OFFLINE OFFLINE

ora.evmd

1        ONLINE  ONLINE       andy_phydg

[grid@andy_phydg ~]$

2：问题

[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp

Executing /data1/grid/crs/perl/bin/perl ./crs/patch11203.pl -patchdir /data1/media/20130820 -patchn psu07 -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp -paramfile /data1/grid/crs/crs/install/crsconfig_params

/data1/grid/crs/crs/install/crsconfig_params

/data1/grid/crs/crs/install/s_crsconfig_defs

This is the main log file: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.report.log

2013-08-25 13:00:37: Starting Oracle Restart Patch Setup

Using configuration parameter file: /data1/grid/crs/crs/install/crsconfig_params

Not able to retreive database home information

[root@andy_phydg OPatch]#

[oracle@andy_phydg 17067783]$ vi /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.log

/bin/su grid -c ' /data1/grid/crs/OPatch/opatch util saveconfigurationsnapshot -configFile /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-00-37.cfg.log '

2013-08-25 13:00:43: /bin/su exited with rc=73

Invoking utility "saveconfigurationsnapshot"

UtilSession failed: UtilSession failed to generate configuration xml file [ PRKH-1010 : Unable to communicate with CRS services.

PRKH-3003 : An attempt to communicate with the CSS daemon failed ]

OPatch failed with error code 73

2013-08-25 13:00:43: Looking for configured databases on node andy_phydg

2013-08-25 13:00:44: Databases configured on node andy_phydg are: PRCR-1119 : Failed to look up CRS resources of database type PRCR-1115 : Failed to find entities of type resource that match filters (TYPE == ora.database.type) and contain attributes DB_UNIQUE_NAME,ORACLE_HOME,VERSION Cannot communicate with crsd

2013-08-25 13:00:44: Determining ORACLE_HOME paths for configured databases

2013-08-25 13:00:44: Executing cmd: /data1/grid/crs/bin/srvctl config database -d PRCR-1119 : Failed to look up CRS resources of database type

2013-08-25 13:00:44: Command output:

>  PRKO-2002 : Invalid command line option: :

>End Command output

2013-08-25 13:00:44: try with srvctl config database -v

2013-08-25 13:00:44: Executing cmd: /data1/grid/crs/bin/srvctl config database -v

2013-08-25 13:00:45: Command output:

>  PRCR-1119 : Failed to look up CRS resources of database type

>  PRCR-1115 : Failed to find entities of type resource that match filters (TYPE == ora.database.type) and contain attributes DB_UNIQUE_NAME,ORACLE_HOME,VERSION

>  Cannot communicate with crsd

>End Command output

2013-08-25 13:00:45: Not able to retreive database home information

3：尝试

[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -oh /data1/grid/crs -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp

Executing /data1/grid/crs/perl/bin/perl ./crs/patch11203.pl -patchdir /data1/media/20130820 -patchn psu07 -oh /data1/grid/crs -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp -paramfile /data1/grid/crs/crs/install/crsconfig_params

/data1/grid/crs/crs/install/crsconfig_params

/data1/grid/crs/crs/install/s_crsconfig_defs

This is the main log file: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-06-05.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-06-05.report.log

2013-08-25 13:06:05: Starting Oracle Restart Patch Setup

Using configuration parameter file: /data1/grid/crs/crs/install/crsconfig_params

Clusterware is either  not running or not configured. You have the following 2 options

1. Configure and Start the Clusterware on this node and re-run the tool

2. or Run the tool with the -och option and then invoke  tool with -oh to patch the RDBMS home

[root@andy_phydg OPatch]#

4：解决

[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -och /data1/grid/crs -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp

patch /data1/media/20130820/psu07/16619898  apply successful for home  /data1/grid/crs

patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/grid/crs

CRS-4123: Oracle High Availability Services has been started.

[root@andy_phydg OPatch]#

停掉has,会Checking the status of ohasd失败

[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -oh /data1/andydb/db -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp

Executing /data1/grid/crs/perl/bin/perl ./crs/patch11203.pl -patchdir /data1/media/20130820 -patchn psu07 -oh /data1/andydb/db -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp -paramfile /data1/grid/crs/crs/install/crsconfig_params

/data1/grid/crs/crs/install/crsconfig_params

/data1/grid/crs/crs/install/s_crsconfig_defs

This is the main log file: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-45-23.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /data1/grid/crs/cfgtoollogs/opatchauto2013-08-25_13-45-23.report.log

2013-08-25 13:45:23: Starting Oracle Restart Patch Setup

Using configuration parameter file: /data1/grid/crs/crs/install/crsconfig_params

Clusterware is either  not running or not configured. You have the following 2 options

1. Configure and Start the Clusterware on this node and re-run the tool

2. or Run the tool with the -och option and then invoke  tool with -oh to patch the RDBMS home

[root@andy_phydg OPatch]#

启动has

[root@andy_phydg OPatch]# ./opatch auto /data1/media/20130820/psu07 -oh /data1/andydb/db -ocmrf /data1/grid/crs/OPatch/ocm/bin/ocm.rsp

patch /data1/media/20130820/psu07/16619898/custom/server/16619898  apply successful for home  /data1/andydb/db

patch /data1/media/20130820/psu07/16619892  apply successful for home  /data1/andydb/db

[root@andy_phydg OPatch]#

DB不在crs里，需要sqlplus到mount standby database去开启mrp。