记一次jvm crash

最新推荐文章于 2021-08-04 15:34:42 发布

pwlazy

最新推荐文章于 2021-08-04 15:34:42 发布

阅读量9.2k

点赞数

分类专栏： java 文章标签： crash jvm velocity byte java 网络

本文链接：https://blog.csdn.net/pwlazy/article/details/7733945

版权

java 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

昨天下午4：30，系统发生了有史以来最刺激的崩溃，一开始是２台挂了（线上共５台），这个基本上很少见，一般情况下都是偶尔一台挂掉。

赶紧手动重启，搞笑的是重启起来的机器里马也挂了，结果就出现了一种尴尬的局面，我们不停的重启，机器不断地报警，不断地挂，整个过程持续了20多分钟。

其实上个礼拜系统就挂过两次，第１次挂了无任何信息没有出现常见的hs*.log这样的jvm crash日志，cpu、内存、流量等指标也没有异常，打算打开ulimit -c 获取系统core,还没弄上去第二次crash又发生了，于是把所有能看得日志都查了一遍，也在/var/log/messages中找到了

"Jul 5 10:25:46 cms-web1 kernel:java[24016]: segfault at 00000000457ebd28 rip 00002aaab75a35cd rsp 00000000457ebd10 error 6"

臭名昭著的segfault出现了，于是赶紧在每台机器加上了ulimit -c unlimited

20分钟的忙碌终于过去了，于是赶紧分析core日志

gdb java core-java-**.log

看到重要信息

Core was generated by `/usr/xx/java/bin/java -Dprogram.name=run.sh -server -Xmx2g -Xms2g -Xmn512m'.

Program terminated with signal 11, Segmentation fault.

接下来通过gdb的bt看栈信息

(gdb) bt
#0  0x00002aaab6cc75cd in Java_java_net_SocketOutputStream_socketWrite0 () from /usr/xx/install/jdk1.6.0_25/jre/lib/amd64/libnet.so
#1  0x00002aaaab041b0c in ?? ()
#2  0x0000000000000000 in ?? ()

只能看到最后在访问网络的时候挂了，但栈信息太少了

于是通过jstack /path/to/java coredumpfile 终于看到了有用的信息

Thread 26089: (state = IN_NATIVE)

- java.net.SocketOutputStream.socketWrite0(java.io.FileDescriptor, byte[], int, int) @bci=0 (Interpreted frame)

- java.net.SocketOutputStream.socketWrite(byte[], int, int) @bci=44, line=92 (Interpreted frame)

- java.net.SocketOutputStream.write(byte[], int, int) @bci=4, line=136 (Interpreted frame)

- oracle.net.ns.DataPacket.send(int) @bci=50 (Interpreted frame)

- oracle.net.ns.NetOutputStream.flush() @bci=15 (Interpreted frame)

- oracle.net.ns.NetInputStream.getNextPacket() @bci=41 (Interpreted frame)

- oracle.net.ns.NetInputStream.read(byte[], int, int) @bci=33 (Compiled frame)

- oracle.net.ns.NetInputStream.read(byte[]) @bci=5 (Compiled frame)

- oracle.net.ns.NetInputStream.read() @bci=5 (Compiled frame)

- oracle.jdbc.driver.T4CMAREngine.unmarshalUB1() @bci=6, line=1099 (Compiled frame)

- oracle.jdbc.driver.T4CMAREngine.unmarshalSB1() @bci=1, line=1070 (Interpreted frame)

- oracle.jdbc.driver.T4C8Oall.receive() @bci=52, line=478 (Interpreted frame)

- oracle.jdbc.driver.T4CPreparedStatement.doOall8(boolean, boolean, boolean, boolean) @bci=619, line=216 (Interpreted frame)

- oracle.jdbc.driver.T4CPreparedStatement.executeForRows(boolean) @bci=16, line=955 (Interpreted frame)

- oracle.jdbc.driver.OracleStatement.executeMaybeDescribe() @bci=154, line=1060 (Interpreted frame)

- oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe() @bci=11, line=839 (Interpreted frame)

- oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout() @bci=132, line=1132 (Interpreted frame)

- oracle.jdbc.driver.OraclePreparedStatement.executeInternal() @bci=94, line=3316 (Interpreted frame)

- oracle.jdbc.driver.OraclePreparedStatement.executeQuery() @bci=17, line=3361 (Interpreted frame)

- com.xx.druid.filter.FilterChainImpl.preparedStatement_executeQuery(com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=29, line=2564 (Interpreted frame)

- com.xx.druid.filter.FilterAdapter.preparedStatement_executeQuery(com.xx.druid.filter.FilterChain, com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=2, line=931 (Interpreted frame)

- com.xx.druid.filter.FilterEventAdapter.preparedStatement_executeQuery(com.xx.druid.filter.FilterChain, com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=14, line=459 (Interpreted frame)

- com.xx.druid.filter.FilterChainImpl.preparedStatement_executeQuery(com.xx.druid.proxy.jdbc.PreparedStatementProxy) @bci=17, line=2561 (Interpreted frame)

- com.xx.druid.proxy.jdbc.PreparedStatementProxyImpl.executeQuery() @bci=5, line=83 (Interpreted frame)

- org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeQuery() @bci=4, line=90 (Interpreted frame)

- org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery() @bci=12, line=236 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.resourceloader.PreviewTemplateResourceLoader.getResourceStream(java.lang.String) @bci=80, line=65 (Interpreted frame)

- org.apache.velocity.Template.process() @bci=20, line=98 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.resourceloader.PreviewResourceManagerImpl.loadResource(java.lang.String, int, java.lang.String) @bci=115, line=116 (Interpreted frame)

- org.apache.velocity.runtime.resource.ResourceManagerImpl.getResource(java.lang.String, int, java.lang.String) @bci=132, line=354 (Interpreted frame)

- org.apache.velocity.runtime.RuntimeInstance.getTemplate(java.lang.String, java.lang.String) @bci=11, line=1400 (Interpreted frame)

- org.apache.velocity.runtime.RuntimeInstance.getTemplate(java.lang.String) @bci=6, line=1380 (Interpreted frame)

- org.apache.velocity.app.VelocityEngine.getTemplate(java.lang.String) @bci=5, line=401 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.DefaultVelocityTemplateEngine.mergeTemplate(org.apache.velocity.context.Context, java.lang.String, int) @bci=15, line=415 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.DefaultVelocityTemplateEngine.previewTemplate(java.util.Map, java.lang.String, int) @bci=13, line=310 (Interpreted frame)

- com.xx.china.cmshollywood.cms.biz.tdprender.OfferRender.getOfferAndTradeInfoByOfferIds(java.lang.String, java.lang.String, java.lang.String, java.lang.String) @bci=129, line=275 (Interpreted frame)

- sun.reflect.GeneratedMethodAccessor283.invoke(java.lang.Object, java.lang.Object[]) @bci=64 (Interpreted frame)

- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)

- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Compiled frame)

- org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(java.lang.Object, java.lang.Object[]) @bci=6, line=389 (Interpreted frame)

- org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=66, line=378 (Interpreted frame)

- org.apache.velocity.runtime.parser.node.ASTMethod.execute(java.lang.Object, org.apache.velocity.context.InternalContextAdapter) @bci=449, line=270 (Interpreted frame)

- org.apache.velocity.runtime.parser.node.ASTReference.execute(java.lang.Object, org.apache.velocity.context.InternalContextAdapter) @bci=192, line=252 (Compiled frame)

- org.apache.velocity.runtime.parser.node.ASTReference.render(org.apache.velocity.context.InternalContextAdapter, java.io.Writer) @bci=30, line=332 (Interpreted frame)

- org.apache.velocity.runtime.parser.node.SimpleNode.render(org.apache.velocity.context.InternalContextAdapter, java.io.Writer) @bci=21, line=336 (Interpreted frame)

- org.apache.velocity.Template.merge(org.apache.velocity.context.Context, java.io.Writer, java.util.List) @bci=251, line=328 (Interpreted frame)

- org.apache.velocity.Template.merge(org.apache.velocity.context.Context, java.io.Writer) @bci=4, line=235 (Interpreted frame)

- com.xx.china.cmshollywood.cms.engine.DefaultVelocityTemplateEngine.mergeTemplate(org.apache.velocity.context.Context, java.lang.String, int) @bci=25, line=416 (Interpreted frame)