1,Tomcat启动报错例如以下:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memcachedClient' defined in
file [/usr/local/apache-tomcat-6.0.37_6500/webapps/trade_service/WEB-INF/classes/META-INF/spring/springContext_memcached.xml]:
Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method
[public net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException]
threw exception; nested exception is java.lang.OutOfMemoryError
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod
(ConstructorResolver.java:581)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod
(AbstractAutowireCapableBeanFactory.java:1015)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance
(AbstractAutowireCapableBeanFactory.java:911)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean
(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean
(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton
(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference
(BeanDefinitionValueResolver.java:322)
... 44 more
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public
net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException] threw
exception; nested exception is java.lang.OutOfMemoryError
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate
(SimpleInstantiationStrategy.java:169)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod
(ConstructorResolver.java:570)
... 53 more
Caused by: java.lang.OutOfMemoryError
,
2,看到创建memcache clent报错,立即去memcacheserver查看一下,memcache正常执行,内存也足够剩余。
[root@localhost ~]# tail -f /tmp/memcache_error.log
<287 0x00 0x00 0x00 0x00
<287 0x00 0x00 0x00 0x00
<287 0x00 0x00 0x00 0x00
>287 Writing bin response:
>287 0x81 0x0b 0x00 0x00
>287 0x00 0x00 0x00 0x00
>287 0x00 0x00 0x00 0x06
>287 0x00 0x00 0x00 0x00
>287 0x00 0x00 0x00 0x00
>287 0x00 0x00 0x00 0x00
...
[root@localhost ~]# free -m
total used free shared buffers cached
Mem: 5938 3368 2570 0 195 296
-/+ buffers/cache: 2876 3061
Swap: 7935 0 7935
[root@localhost ~]#
问题在哪里?再看看日志里面有误error或者warning信息:
[root@localhost ~]# cat /tmp/memcache_error.log |grep warning
[root@localhost ~]# cat /tmp/memcache_error.log |grep error
[root@localhost ~]#
一点错误信息也没有,推断问题不是出在memcache这一块。3,去check tomcatserver
[root@localhost logs]# free -m
total used free shared buffers cached
Mem: 18022 16943 1078 0 104 675
-/+ buffers/cache: 16162 1859
Swap: 20031 8012 12019
[root@localhost logs]#
仅仅剩余1个G的内存,去改动一下tomcat的内存配置:
[root@localhost bin]# vim catalina.sh
#JAVA_OPTS="-Xms2048m -Xmx2048m -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=768m"
# add by wangqi on 20140614
JAVA_OPTS="-Xms512m -Xmx512m -Xss128K -XX:PermSize=64m -XX:MaxPermSize=256m"
再重新启动tomcat,看到还是报一样的错误。
4,回到应用本身去查问题
跟开发者一起排查,看到是trade_service启动了过多的多余的事务,把tomcat的内存耗光了,所以关闭掉多余的事务,再启动tomcat,就不会再报错。
感慨:就跟大部分web应用卡,页面登录不上去,对应过慢非常多是出在效率低下的sql上面,tomcat启动不起来,内存溢出大部分也是在java代码上面,大家须要写代码或者sql须要慎重,且写且慎重。