Rac opatch小补丁之后其中一实例不能启动

Normal 0 7.8 磅 0 2 false false false EN-US ZH-CN X-NONE MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:普通表格; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Calibri","sans-serif";}

1：停止rac的所有实例

2：opatch补丁5442780 在prod1上，正常完成

3：opatch补丁5442780在prod2上，正常完成

4：startup prod1正常

5：startup prod2失败

SQL> startup

ORA-03113: end-of-file on communication channel

TRACE FILE:

Fri Oct 10 23:41:59 2008

Errors in file /u01/oracle/10.2.0/admin/PROD2_erpdb2/udump/prod2_ora_24163.trc:

ORA-00600: internal error code, arguments: [1403], [], [], [], [], [], [], []

Fri Oct 10 23:42:02 2008

Errors in file /u01/oracle/10.2.0/admin/PROD2_erpdb2/bdump/prod2_ckpt_24297.trc:

ORA-00081: address range [0x60000000000A7E60, 0x60000000000A7E64) is not readable

ORA-07445: exception encountered: core dump [$cold_kslgetsl()+1424] [SIGSEGV] [Address not mapped to object] [0x3FFFFFFF812F7E52] []

[]

Fri Oct 10 23:42:04 2008

Errors in file /u01/oracle/10.2.0/admin/PROD2_erpdb2/bdump/prod2_pmon_24249.trc:

ORA-00469: CKPT process terminated with error

Fri Oct 10 23:42:04 2008

PMON: terminating instance due to error 469

Fri Oct 10 23:42:04 2008

Errors in file /u01/oracle/10.2.0/admin/PROD2_erpdb2/bdump/prod2_lms1_24289.trc:

ORA-00469: CKPT process terminated with error

Fri Oct 10 23:42:04 2008

Errors in file /u01/oracle/10.2.0/admin/PROD2_erpdb2/bdump/prod2_lms0_24287.trc:

ORA-00469: CKPT process terminated with error

Fri Oct 10 23:42:04 2008

System state dump is made for local instance

System State dumped to trace file /u01/oracle/10.2.0/admin/PROD2_erpdb2/bdump/prod2_diag_24251.trc

Fri Oct 10 23:42:04 2008

Trace dumping is performing id=[cdmp_20081010234204]

Fri Oct 10 23:42:04 2008

Errors in file /u01/oracle/10.2.0/admin/PROD2_erpdb2/bdump/prod2_diag_24251.trc:

ORA-07445: exception encountered: core dump [T_19_2d92_cl___doprnt_main()+35296] [SIGSEGV] [Address not mapped to object] [0x6C61746

3683A206F] [] []

Fri Oct 10 23:42:08 2008

Errors in file /u01/oracle/10.2.0/admin/PROD2_erpdb2/bdump/prod2_diag_24251.trc:

ORA-00081: address range [0x60000000000A7E60, 0x60000000000A7E64) is not readable

ORA-07445: exception encountered: core dump [T_19_2d92_cl___doprnt_main()+35296] [SIGSEGV] [Address not mapped to object] [0x6C61746

3683A206F] [] []

6：停止所有实例，opatch rollback -id 5442780 （prod2）

7：重新apply 5442780 （prod2），启动仍然失败

8：查看metalink doc Note:377437.1 Oracle Instance does not startup on one of RAC the nodes after installing any One-off Patch

Solution

Copy the good library in the faulty node and manually re-link the Oracle again.

Ensure all instances are shut down !

Manually copy good $ORACLE_HOME/rdbms/lib/libknlopt.a from the local node to all remote faulty nodes and relink Oracle again on all faulty nodes.

eg: cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ioracle

9：重新apply 5442780 （prod2），正常完成，启动正常

10：启动所有实例以及applications

11：访问apps正常，sqlplus连接db正常

Oracle Instance does not startup on one of RAC the nodes after installing any One-off Patch

Doc ID: Note:377437.1 Type: PROBLEM

Last Revision Date: 13-APR-2008 Status: MODERATED

$ unzip p5442780_10202_HPUX-IA64.zip

Archive: p5442780_10202_HPUX-IA64.zip

creating: 5442780/

creating: 5442780/files/

creating: 5442780/files/lib/

creating: 5442780/files/lib/libserver10.a/

inflating: 5442780/files/lib/libserver10.a/qerfx.o

creating: 5442780/etc/

creating: 5442780/etc/config/

inflating: 5442780/etc/config/inventory

inflating: 5442780/etc/config/actions

creating: 5442780/etc/xml/

inflating: 5442780/etc/xml/GenericActions.xml

inflating: 5442780/etc/xml/ShiphomeDirectoryStructure.xml

inflating: 5442780/README.txt

$ ls

5292883 arch_erpdb2 oracle tmp

5442780 backup_data p5292883_10202_HPUX-IA64.zip

arch_erpdb1 lost+found p5442780_10202_HPUX-IA64.zip

$ cd 54*

$ ls

README.txt etc files

$ pwd

/u01/5442780

$ /u01/oracle/10.2.0/OPatch/opatch apply

Invoking OPatch 10.2.0.2.0

Oracle interim Patch Installer version 10.2.0.2.0

Copyright (c) 2005, Oracle Corporation. All rights reserved..

Oracle Home : /u01/oracle/10.2.0

Central Inventory : /u01/oracle/oraInventory

from : /u01/oracle/10.2.0/oraInst.loc

OPatch version : 10.2.0.2.0

OUI version : 10.2.0.2.0

OUI location : /u01/oracle/10.2.0/oui

Log file location : /u01/oracle/10.2.0/cfgtoollogs/opatch/opatch-00_Oct_10_23-26-42-GMT+08_Fri.log

ApplySession applying interim patch '5442780' to OH '/u01/oracle/10.2.0'

Subset patches: 5442780

The fixes for Patch 5442780 are included in the patch currently being installed (5442780). If you continue, they will be rolled back and the new patch (5442780) will be installed.

Do you want to proceed? [y|n]

y

User Responded with: Y

Invoking fuser to check for active processes.

Invoking fuser on "/u01/oracle/10.2.0/bin/oracle"

OPatch will patch the local system only because this is a shared Oracle Home.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/oracle/10.2.0')

Is the local system ready for patching?

Do you want to proceed? [y|n]

y

User Responded with: Y

Backing up files and inventory (not for auto-rollback) for the Oracle Home

Backing up files affected by the patch '5442780' for restore. This might take a while...

Backing up files affected by the patch '5442780' for restore. This might take a while...

ApplySession rolling back interim patch '5442780' from OH '/u01/oracle/10.2.0'

Patching component oracle.rdbms, 10.2.0.2.0...

Updating archive file "/u01/oracle/10.2.0/lib/libserver10.a" with "lib/libserver10.a/qerfx.o"

Running make for target ioracle

RollbackSession removing interim patch '5442780' from inventory

OPatch back to application of the patch '5442780' after auto-rollback.

Backing up files affected by the patch '5442780' for rollback. This might take a while...

Patching component oracle.rdbms, 10.2.0.2.0...

Updating archive file "/u01/oracle/10.2.0/lib/libserver10.a" with "lib/libserver10.a/qerfx.o"

Running make for target ioracle

ApplySession adding interim patch '5442780' to inventory

Verifying the update...

Inventory check OK: Patch ID 5442780 is registered in Oracle Home inventory with proper meta-data.

Files check OK: Files from Patch ID 5442780 are present in Oracle Home.

The local system has been patched and can be restarted.

OPatch succeeded.

$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.2.0 - Production on Fri Oct 10 23:31:08 2008

Copyright (c) 1982, 2005, Oracle. All Rights Reserved.

staConnected to an idle instance.

SQL> rtup

ORA-03113: end-of-file on communication channel

SQL> shutdown immediate

ORA-01034: ORACLE not available

ORA-27101: shared memory realm does not exist

HPUX-ia64 Error: 2: No such file or directory

SQL> exit

pwDisconnected

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/594850/viewspace-578012/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/594850/viewspace-578012/