环境:AIX 6.1、EBS R12、DB 11.1.0.7
克隆EBS数据库,在数据库数据恢复完毕后做DB层的autoconfig操作,出现卡住的现象,并到最后出现error报错,但没有发现有比较明确的报错信息。
由于DB层的autoconfig失败,决定做一遍DB层的配置恢复
$cd /ebsdev_bp/DEV/db/tech_st/11.1.0/appsutil/clone/bin
$ perl adcfgclone.pl dbTechStack
执行成功,随即启动数据库进行APP层的恢复
$cd /tmpfs/DEV/apps/apps_st/comn/clone/bin
$perl adcfgclone.pl appsTier
执行到50%再次出现卡住的情况,查询相关后台日志看到卡在下面这个脚本:
AutoConfig Setup Phase
Running Setup Process 2 of 7 for AD_TOP
Executing script in InstantiateFile:
/tmpfs/DEV/inst/apps/DEV_dbtest/admin/install/adgendbc.sh
对整个日志文件都看了一遍,还发现有如下报错:
Timed out( 300000 ): Interrupted Exception
##########################################################################
Generate Tns Names
##########################################################################
Logfile: /tmpfs/DEV/inst/apps/DEV_dbtest/admin/log/NetServiceHandler.log
Classpath : /tmpfs/DEV/apps/apps_st/comn/java/lib/appsborg2.zip:/tmpfs/D
EV/apps/apps_st/comn/java/classes
Updating s_tnsmode to ‘generateTNS’
UpdateContext exited with status: 0
说明有超时的存在
如果当卡在50%的时候不手工结束掉会卡1个多小时后报ERROR自动退出,类似如下报错:
Beginning application tier Apply – Mon Jul 15 13:41:42 2013
/tmpfs/DEV/apps/apps_st/comn/clone/bin/../jre/bin/java -Xmx600M -DCONTEXT_VALIDA
TED=true -Doracle.installer.oui_loc=/oui -classpath /tmpfs/DEV/apps/apps_st/com
n/clone/jlib/xmlparserv2.jar:/tmpfs/DEV/apps/apps_st/comn/clone/jlib/ojdbc14.jar
:/tmpfs/DEV/apps/apps_st/comn/clone/jlib/java:/tmpfs/DEV/apps/apps_st/comn/clone
/jlib/oui/OraInstaller.jar:/tmpfs/DEV/apps/apps_st/comn/clone/jlib/oui/ewt3.jar:
/tmpfs/DEV/apps/apps_st/comn/clone/jlib/oui/share.jar:/tmpfs/DEV/apps/apps_st/co
mn/clone/jlib/oui/srvm.jar:/tmpfs/DEV/apps/apps_st/comn/clone/jlib/ojmisc.jar o
racle.apps.ad.clone.ApplyAppsTier -e /tmpfs/DEV/inst/apps/DEV_dbtest/appl/admin/
DEV_dbtest.xml -stage /tmpfs/DEV/apps/apps_st/comn/clone -showProgress
APPS Password : Log file located at /tmpfs/DEV/inst/apps/DEV_dbtest/admin/log/Ap
plyAppsTier_07151341.log
/ 71% completed
\ 90% completed
ERROR while running Apply…
Mon Jul 15 15:47:43 2013
ERROR: Failed to execute /tmpfs/DEV/apps/apps_st/comn/clone/bin/adclone.pl
Please check logfile.
经后台日志查询,又发现有相关超时报错:
adgendbc.sh exiting with status 1
ERRORCODE = 1 ERRORCODE_END
.end std out.
.end err out.
相关adgendbc.sh脚本执行失败
ERROR
———————–
AutoConfig is exiting with status 30
RC-50014: Fatal: Execution of AutoConfig was failed
Raised by oracle.apps.ad.clone.ApplyApplTop
ERROR: AutoConfig completed with errors. Check logfile at /tmpfs/DEV/inst/apps/DEV_dbtest/admin/log/ApplyAppsTier_07161733.log for details.
ApplyApplTop Completed Successfully.
由于刚刚克隆完DB后看到后台日志存在许多CLIENT端超时的信息,故再次检查ALERT日志信息,存在如下超时报错:
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
Oracle Bequeath NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
Time: 16-JUL-2013 22:56:58
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.18.69)(PORT=65244))
WARNING: inbound connection timed out (ORA-3136)
Tue Jul 16 22:57:08 2013
***********************************************************************
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
Oracle Bequeath NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
Time: 16-JUL-2013 22:57:08
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.18.66)(PORT=17951))
WARNING: inbound connection timed out (ORA-3136)
Tue Jul 16 22:57:38 2013
***********************************************************************
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
Oracle Bequeath NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.1.0.7.0 – Production
Time: 16-JUL-2013 22:57:38
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.18.80)(PORT=54729))
WARNING: inbound connection timed out (ORA-3136)
Tue Jul 16 22:57:48 2013
经应用确认,以前该套数据库上部署了WEBLOGIC服务,并且正好是在80、69、66上,随即把相关WEBLOGIC服务关闭,发现数据库后台不再报错。
再次对DB做autoconfig操作
执行成功返回successfully的信息。
再次在应用端执行perl adcfgclone.pl appsTier,执行成功,没有出现超时的现象。
基本确定WEBLOGIC影响了EBS的克隆操作。