APPLIES TO:
Oracle Database - Enterprise Edition - Version 12.1.0.2 and laterIBM AIX on POWER Systems (64-bit)
SYMPTOMS
Applying patch 19582630 DB SERVICES ARE AUTOMATICALLY BOUNCED DUE TO MEMORY GUARD to 12.1.0.2 Grid Infrastructure HOME on AIX platform following README fails at "opatch napply" with:
$ /haclu/64bit/crs121/OPatch/opatch napply -local -oh /haclu/64bit/crs121 -id 19582630
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2015, Oracle Corporation. All rights reserved.
Oracle Home : /haclu/64bit/crs121
Central Inventory : /haclu/64bit/crs121/oraInventory
from : /haclu/64bit/crs121/oraInst.loc
OPatch version : 12.1.0.1.3
OUI version : 12.1.0.2.0
Log file location : /haclu/64bit/crs121/cfgtoollogs/opatch/opatch2015-05-19_05-44-34AM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 19582630
......
Patching component oracle.wlm.dbwlm, 12.1.0.2.0...
ApplySession failed in system modification phase... 'ApplySession::apply failed: + PATH=/bin:/usr/bin:/usr/ccs/bin
+ export PATH
+ lib=/haclu/64bit/crs121/lib/libasmclntsh12.so
+ makefile=/haclu/64bit/crs121/rdbms/lib/ins_rdbms.mk
+ so_ext=so
+ target=ld_shlib
+ + basename /haclu/64bit/crs121/lib/libasmclntsh12.so .so
libname=libasmclntsh12
+ + dirname /haclu/64bit/crs121/lib/libasmclntsh12.so
sodir=/haclu/64bit/crs121/lib
+ ardir=/haclu/64bit/crs121/lib/
+ [ var = ld_shlib ]
+ suffix=LIBS
+ var=
+ [ ! -f /haclu/64bit/crs121/lib/libasmclntsh12.a ]
+ [ != ]
+ make -f /haclu/64bit/crs121/rdbms/lib/ins_rdbms.mk ld_shlib _FULL_LIBNAME=/haclu/64bit/crs121/lib/libasmclntsh12.so _LIBNAME=libasmclntsh12 _LIBDIR=/haclu/64bit/crs121/lib/ _LIBNAME_LIBS=$(libasmclntsh12LIBS) _LIBNAME_EXTRALIBS=$(libasmclntsh12EXTRALIBS)
ld: 0706-003 Cannot find or read import file: /haclu/64bit/crs121/rdbms/admin/libasmclntsh12.imp
ld:accessx(): No such file or directory
make: The error code from the last command is 255.
......
Log file location: /haclu/64bit/crs121/cfgtoollogs/opatch/opatch2015-05-19_05-44-34AM_1.log
OPatch failed with error code 73
After that CRS can not be started via "rootcrs.pl -patch" or "crsctl start crs" :
Using configuration parameter file: /haclu/64bit/crs121/crs/install/crsconfig_params
Could not load program /haclu/64bit/crs121/bin/crsctl.bin:
Dependent module libasmclntsh12.so could not be loaded.
Could not load module libasmclntsh12.so.
System error: No such file or directory
Died at /haclu/64bit/crs121/crs/install/crspatch.pm line 805.
CHANGES
New installation of 12.1.0.2 without any PSU applied.
CAUSE
The missing file libasmclntsh12.imp is due to unpublished BUG 20117253 - UNSATISFIEDLINKERROR FOR LIBPATCHGENSH12.SO ON 121020 AIX PLATFORM
This bug is fixed in 12.1.0.2.2 PSU
The library libasmclntsh12.so was removed as part of patch apply, due to the relink failure, libasmclntsh12.so was not re-generated, causing further CRS startup failure.
SOLUTION
1. Copy the file libasmclntsh12.so under GRID_HOME/lib from a working node (racnode2) to the problem node (racnode1) as root user:
# scp /haclu/64bit/crs121/lib/libasmclntsh12.so root@racnode1:/haclu/64bit/crs121/lib/libasmclntsh12.so
This file should be owned by root:oinstall with permission 644:
-rw-r--r-- 1 root oinstall 1826909 May 14 07:56 /haclu/64bit/crs121/lib/libasmclntsh12.so
2. As root user, run rootcrs.pl -patch to restart the GI stack:
# /haclu/64bit/crs121/crs/install/rootcrs.pl -patch
3. Download patch 20117253 for 12.1.0.2.0, apply it to GI_HOME following the README .
4. If you want apply two patches with one GI stop and restart, after step 2 in patch 20117253 README
As the GI home owner execute:
$ <GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/20117253
apply patch 19582630
$ <GI_HOME>/OPatch/opatch napply -local -oh <GI_HOME> -id 19582630
Then continue with step 3 Run the post script.
5. Once above patch applied successfully, opatch lsinventory should show 2 patches applied:
Interim patches (2) :
Patch 19582630 : applied on Tue May 19 06:29:24 UTC 2015
Unique Patch ID: 18359222
Patch description: "WLM Patch Set Update: 12.1.0.2.0 (19582630)"
Created on 21 Jan 2015, 07:14:28 hrs PST8PDT
Bugs fixed:
19582630
Patch 20117253 : applied on Tue May 19 06:21:56 UTC 2015
Unique Patch ID: 18406385
Created on 24 Dec 2014, 02:39:39 hrs PST8PDT
Bugs fixed:
20117253
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -------------
0 racnode2
1378019997 racnode1
--------------------------------------------------------------------------------
OPatch succeeded.
6. Continue with other nodes patching.