weblogic堆积的数据库链接1

   前几天分析一份内存溢出的dump文件，发现堆积了很多数据库连接，多的有17M，少的有1K,完全不知道原因。今天碰到一个类似的问题，幸运的是把JVM的thread dump拿回来了，用IBM Thread and monitor Dump Analyzer for Java分析可以得出，是一个数据库连接的操作慢导致341个数据库连接等待。由于不能贴图，就贴一个日志：

   锁其实就是一个内存地址，下面的两个线程都在要锁<0x0000000701541340>。

"[ACTIVE] ExecuteThread: '400' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f6830238000 nid=0x16121 waiting for monitor entry [0x00007f68849c8000]

   java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:342)
- waiting to lock <0x0000000701541340> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.jta.DataSource.end(DataSource.java:953)
at weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1306)
at weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:424)
at weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:344)
at weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1689)
at weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2082)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3238)
at weblogic.transaction.internal.ServerTransactionImpl.access$100(ServerTransactionImpl.java:59)
at weblogic.transaction.internal.ServerTransactionImpl$1.run(ServerTransactionImpl.java:3480)
at weblogic.transaction.internal.PlatformHelperImpl$3.run(PlatformHelperImpl.java:1052)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:312)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:264)

"[ACTIVE] ExecuteThread: '393' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f683022b800 nid=0x15ede waiting for monitor entry [0x00007f68850cf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:342)
- waiting to lock <0x0000000701541340> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.jta.DataSource.end(DataSource.java:953)
at weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1306)
at weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:424)
at weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:344)
at weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1689)
at weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2082)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3238)
at weblogic.transaction.internal.ServerTransactionImpl.access$100(ServerTransactionImpl.java:59)
at weblogic.transaction.internal.ServerTransactionImpl$1.run(ServerTransactionImpl.java:3480)
at weblogic.transaction.internal.PlatformHelperImpl$3.run(PlatformHelperImpl.java:1052)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:312)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:264)

 幕后黑手找到了，是这个线程锁住了locked <0x0000000701541340>
"[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f6830028000 nid=0x3fd6 runnable [0x00007f689e8e4000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at oracle.net.ns.Packet.receive(Packet.java:300)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.jdbc.driver.T4CMAREngine.unmarshalNBytes(T4CMAREngine.java:1435)
at oracle.jdbc.driver.T4CTTIOtxse.readRPA(T4CTTIOtxse.java:304)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:304)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4CTTIOtxse.doOTXSE(T4CTTIOtxse.java:163)
at oracle.jdbc.driver.T4CXAResource.doEnd(T4CXAResource.java:358)
     - locked <0x0000000701541340> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:413)
     - locked <0x0000000701541340> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.jta.DataSource.end(DataSource.java:953)
at weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1306)
at weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:424)
at weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:344)
at weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1689)
at weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2082)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3238)
at weblogic.transaction.internal.ServerTransactionImpl.globalRollback(ServerTransactionImpl.java:2987)
at weblogic.transaction.internal.ServerTransactionImpl.internalRollback(ServerTransactionImpl.java:425)
at weblogic.transaction.internal.ServerTransactionImpl.rollback(ServerTransactionImpl.java:403)
at org.glassfish.transaction.TransactionManagerImplCommon.rollback(TransactionManagerImplCommon.java:551)
at org.springframework.transaction.jta.JtaTransactionManager.doRollback(JtaTransactionManager.java:1038)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:846)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:823)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:493)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:264)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.comtop.sproc.core.exceptionhandler.aop.SprocFacadeExceptionAspect.doAround(SprocFacadeExceptionAspect.java:45)
at sun.reflect.GeneratedMethodAccessor572.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
at com.gg.app.device.assetchange.facade.internal.AssetchangeInfoFacade$$EnhancerByCGLIB$$2d28236f.downloadAssetchangeInfo(<generated>)
at com.gg.device.assetchange.controller.AssetchangeController.downloadAssetchangeInfo(AssetchangeController.java:689)
at com.gg.device.assetchange.controller.AssetchangeController$$FastClassByCGLIB$$c365d517.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:698)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.gg.monitor.MonitorAspect.aroundAllMethod(MonitorAspect.java:87)
at sun.reflect.GeneratedMethodAccessor583.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.comtop.sproc.core.exceptionhandler.aop.SprocFacadeExceptionAspect.doAround(SprocFacadeExceptionAspect.java:45)
at sun.reflect.GeneratedMethodAccessor572.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
at com.gg.device.assetchange.controller.AssetchangeController$$EnhancerByCGLIB$$a6465b4b.downloadAssetchangeInfo(<generated>)
at sun.reflect.GeneratedMethodAccessor14099.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:341)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.comtop.top.sys.login.action.SessionFilter.doFilter(SessionFilter.java:219)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.comtop.sproc.system.filter.SprocRequestFilter.doFilter(SprocRequestFilter.java:62)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.comtop.top.component.app.sna.SessionShareFilter.doFilter(SessionShareFilter.java:114)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3388)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3354)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2238)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2154)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2132)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1564)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:312)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:264)