WAS7 EJB OOM问题

在WAS7.0.0.13版本上分布式发布ejb和web,web和ejb不在一个集群,且不在一个server哦情况下,web调用EJB的时候出现OOM错误,报错信息如下:

[11-3-1 3:32:06:132 CST] 00000021 SystemOut     O ADMINISTRATOR:123456:0:1:undefined:192.168.0.103:CrbGu3tv3gf37HzkT5r3BW7::
[11-3-1 3:32:11:331 CST] 00000021 UserManagerDe E com.xx.xxxxxx.web.UserManagerDefaultImpl loginIn java.lang.RuntimeException: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: 
	>> SERVER (id=4773e3aa, host=zmt400) TRACE START:
	>>    java.lang.OutOfMemoryError
	>>	 at com.ibm.rmi.iiop.CDRReader.readBytesForString(CDRReader.java:2296)
	>>	 at com.ibm.rmi.iiop.CDRReader.readStringOrIndirection(CDRReader.java:489)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_codebase_URL(CDRReader.java:2890)
	>>	 at com.ibm.rmi.iiop.CDRReader.fast_read_value(CDRReader.java:1910)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_value(CDRReader.java:2017)
	>>	 at com.xx.xxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.loginIn__com_xx_xxxxx_privilege_UserInfoInterface__CORBA_WStringValue__CORBA_WStringValue__long__long_long__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue(_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.java:206)
	>>	 at com.xxx.xxxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie._invoke(_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.java:127)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:622)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:475)
	>>	 at com.ibm.rmi.iiop.ORB.process(ORB.java:513)
	>>	 at com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
	>>	 at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2841)
	>>	 at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2714)
	>>	 at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
	>>	 at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
	>>	 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)
	>> SERVER (id=4773e3aa, host=zmt400) TRACE END.

[11-3-1 3:32:11:341 CST] 00000021 BaseServer    E com.xx.xxxxx.web.BaseServer processLogin 登录错误
                                 java.lang.RuntimeException: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: 
	>> SERVER (id=4773e3aa, host=zmt400) TRACE START:
	>>    java.lang.OutOfMemoryError
	>>	 at com.ibm.rmi.iiop.CDRReader.readBytesForString(CDRReader.java:2296)
	>>	 at com.ibm.rmi.iiop.CDRReader.readStringOrIndirection(CDRReader.java:489)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_codebase_URL(CDRReader.java:2890)
	>>	 at com.ibm.rmi.iiop.CDRReader.fast_read_value(CDRReader.java:1910)
	>>	 at com.ibm.rmi.iiop.CDRReader.read_value(CDRReader.java:2017)
	>>	 at com.xx.xxxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxx_common_service_e_d15d50c6_Tie.loginIn__com_xx_xxxxxx_privilege_UserInfoInterface__CORBA_WStringValue__CORBA_WStringValue__long__long_long__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue(_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.java:206)
	>>	 at com.xx.xxxx.common.service.ejb._EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie._invoke(_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.java:127)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:622)
	>>	 at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:475)
	>>	 at com.ibm.rmi.iiop.ORB.process(ORB.java:513)
	>>	 at com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
	>>	 at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2841)
	>>	 at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2714)
	>>	 at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
	>>	 at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
	>>	 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1563)
	>> SERVER (id=4773e3aa, host=zmt400) TRACE END.

 

 

当然第一反应是应用写的有问题,ejb里面有内存泄露或者threadpool开太大,后来检查未发现异常。

 

收集CORE文件和HEAPDUMP文件分析,上PP。

 

core文件分析如下:

 

1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received 
......
......
......
1XMCURTHDINFO  Current Thread Details
NULL           ----------------------
3XMTHREADINFO      "ORB.thread.pool : 0" TID:0x0000000001ECF600, j9thread_t:0x0000000011EA3360, state:R, prio=5
3XMTHREADINFO1            (native thread ID:0x5C7C, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.readBytesForString(CDRReader.java:2279)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.readStringOrIndirection(CDRReader.java:472)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.read_codebase_URL(CDRReader.java:2852)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.fast_read_value(CDRReader.java:1893)
4XESTACKTRACE          at com/ibm/rmi/iiop/CDRReader.read_value(CDRReader.java:2000)
4XESTACKTRACE          at com/xx/xxxxxx/common/service/ejb/_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.loginIn__com_xx_xxxx_privilege_UserInfoInterface__CORBA_WStringValue__CORBA_WStringValue__long__long_long__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue__CORBA_WStringValue(_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie.java:206)
4XESTACKTRACE          at com/xx/xxxxxx/common/service/ejb/_EJSRemoteStatelesscom_xx_xxxxx_common_service_e_d15d50c6_Tie._invoke(_EJSRemoteStatelesscom_xx_xxxxxx_common_service_e_d15d50c6_Tie.java:127)
4XESTACKTRACE          at com/ibm/CORBA/iiop/ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:622)
4XESTACKTRACE          at com/ibm/CORBA/iiop/ServerDelegate.dispatch(ServerDelegate.java:475)
4XESTACKTRACE          at com/ibm/rmi/iiop/ORB.process(ORB.java:504)
4XESTACKTRACE          at com/ibm/CORBA/iiop/ORB.process(ORB.java:1571)
4XESTACKTRACE          at com/ibm/rmi/iiop/Connection.respondTo(Connection.java:2771)
4XESTACKTRACE          at com/ibm/rmi/iiop/Connection.doWork(Connection.java:2640)
4XESTACKTRACE          at com/ibm/rmi/iiop/WorkUnitImpl.doWork(WorkUnitImpl.java:63)
4XESTACKTRACE          at com/ibm/ejs/oa/pool/PooledThread.run(ThreadPool.java:118)
4XESTACKTRACE          at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1527)

 

 

 heapdump分析如下:

 



 

发现CORE文件里面很奇怪,Free Java Heap 有2,104,902,904,98%空闲,但是在内存段分析中,Object即heap占用2,147,483,648 100%占用,heapdump里面分析占用heap内存42,565,544,和core文件的第一种说法匹配,即98% free,排除应用占用内存的可能。

 

分析了下出错的代码行 com.ibm.rmi.iiop.CDRReader.readBytesForString,发现是由web端调用ejb的时候,从ejb容器传递ejb stud的时候出现问题,好,那就避开这个,我将应用发布到同一个server上,这样ejb应该优化为本地调用,验证成功。

 

那为了解决分布式调用的问题,我还要想一个办法,让系统在分布式情况下也不用传递ejb stud,这让我想到了早期中间件需要手动打stud然后引入web容器的事情,查了下infocenter,找到{WAS_HOME}\bin\ejbdeploy.sh命令。

 

ejbdeploy tytim.ear . tytim_stub.ear

 

会在当前目录产生tytim_stub.ear,将这个ear中的ejb相关jar覆盖到.war里面的lib中,问题解决。

 

已联系IBM技术人员,看看是什么原因导致ejb stud传递的时候byte数组长度出现异常,等有了回复,在补充上来。

 

 

转载请注明原始地址: http://boriszhang78.iteye.com/blog/935439

 

 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值