这段时间一直被一个SSH系统性能问题困扰,项目在开发过程中出现Tomcat假死的问题。为了查清系统问题的原因,我找了个SIW软件用来监控系统占用端口,发现数据库连接数时常达到连接池设置的最大连接数。我使用的C3P0连接池,我一开始认为是连接池设置问题,可是问题没有解决。然后开发小组开始修改SSH配置文件,貌似系统偶尔能释放部分数据库连接,可是问题还是没有解决。
后来请教了一技术牛人,我们修改了连接池,把C3P0连接池换成DBCP连接池,并且把连接池配置由hibernate.cfg.xml配置文件中迁移到了Spring配置文件中(为了节省时间,我也就不贴连接池配置的代码了),这下系统能释放大部分连接。可是通过监控发现,保持的数据库连接数还是多于连接池配置的空闲连接数,而且时间一久,系统还是会出现卡死的问题,估计就是连接池保持的连接已经失效导致的。
今天下午,我又上网查了这方面的系统问题的介绍资料,偶尔发现网上介绍JConsole监控JVM和开启线程的相关技术(这里我也不介绍JConsole技术,因为网上实在太多了),了解到JVM设置和数据库连接泄漏都会导致系统问题。因为我对JConsole监控信息是如何反映系统问题不是很了解,所以就将感觉能反映系统性能问题的监控信息贴出来,希望大牛们能明白。
图一 JVM监控信息
图2 内存监控
图3 Tomcat线程监控
Tomcat线程监控中发现【http-8080-XX】线程很有相似性,现在随机贴出两个线程堆栈追踪信息:
名称: http-8080-1
状态:WAITING 在 com.mchange.v2.resourcepool.BasicResourcePool@10d3f0d 上
阻塞总数:9 等待总数: 6
堆栈追踪:
java.lang.Object.wait(Native Method)
com.mchange.v2.resourcepool.BasicResourcePool.awaitAcquire(BasicResourcePool.java:966)
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:208)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:232)
com.mchange.v2.c3p0.PoolBackedDataSource.getConnection(PoolBackedDataSource.java:94)
org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:35)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:417)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:494)
org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:315)
org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:257)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:102)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
$Proxy1.queryDiskList(Unknown Source)
com.iwork.docmgr.action.DiskAction.readDiskInfo(DiskAction.java:59)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:440)
com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:279)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:242)
com.util.LoginInterceptor.intercept(LoginInterceptor.java:24)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:163)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:249)
org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:68)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:122)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:148)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:93)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:235)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:89)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:128)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:104)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:267)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:126)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:138)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:148)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:128)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:52)
org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:468)
org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77)
org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:76)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:174)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:78)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:228)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:216)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:634)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:445)
java.lang.Thread.run(Thread.java:619)
名称: http-8080-23
状态:WAITING 在 com.mchange.v2.resourcepool.BasicResourcePool@10d3f0d 上
阻塞总数:0 等待总数: 2
堆栈追踪:
java.lang.Object.wait(Native Method)
com.mchange.v2.resourcepool.BasicResourcePool.awaitAcquire(BasicResourcePool.java:966)
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:208)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:232)
com.mchange.v2.c3p0.PoolBackedDataSource.getConnection(PoolBackedDataSource.java:94)
org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:35)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:417)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:494)
org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:315)
org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:257)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:102)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
$Proxy26.userLogin(Unknown Source)
com.iwork.basic.action.LoginAction.login(LoginAction.java:78)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:440)
com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:279)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:242)
com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:163)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:249)
org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:68)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:122)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:148)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:93)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:235)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:89)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:128)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:104)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:267)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:126)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:138)
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:148)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:128)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176)
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:236)
org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:52)
org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:468)
org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77)
org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:76)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:174)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:78)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:228)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:216)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:634)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:445)
java.lang.Thread.run(Thread.java:619)
请求大牛们给出紧急援助!!!