一次堆外OOM问题排查

最新推荐文章于 2024-09-26 22:23:24 发布

骚年编程去

最新推荐文章于 2024-09-26 22:23:24 发布

阅读量6.5k

点赞数

分类专栏： JAVA之美文章标签： java 内存泄露

本文链接：https://blog.csdn.net/a837199685/article/details/75784279

版权

本文介绍了线上服务因DirectByteBuffer导致的堆外内存溢出问题的排查过程。通过监控指标发现大量OOM异常，指出Jetty和ElasticSearch客户端由于未关闭连接，产生大量无法回收的DirectByteBuffer，导致Native Memory耗尽。同时分析了-XX:+DisableExplicitGC参数如何影响垃圾回收，以及在特定条件下可能导致的OOM情况。提出了避免此类问题的注意事项。

摘要由CSDN通过智能技术生成

背景

线上服务有一台机器访问不通（一个管理平台),在公司的服务治理平台上查看服务的状况是正常的，说明进程还在。进程并没有完全crash掉。去线上查看机器日志，发现了大量的OOM异常:

017-03-15 00:00:00.041 [WARN] qtp1947699202-120772 nio handle failed
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658) ~[?:1.7.0_76]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.7.0_76]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[?:1.7.0_76]
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) ~[?:1.7.0_76]
        at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~[?:1.7.0_76]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.7.0_76]
        at org.eclipse.jetty.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:235) ~[jetty-io-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.fill(SelectChannelEndPoint.java:326) ~[jetty-io-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1040) ~[jetty-http-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280) ~[jetty-http-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) ~[jetty-http-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) ~[jetty-server-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) [jetty-io-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) [jetty-io-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.10.v20130312.jar:8.1.10.v20130312]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.10.v20130312.jar:8.1.10.v20130312]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_76]

2017-03-15 00:00:00.718 [WARN] elasticsearch[Thundra][transport_client_worker][T#6]{New I/O worker #6} AbstractNioSelector Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658) ~[?:1.7.0_76]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.7.0_76]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[?:1.7.0_76]
        at org.jboss.netty.channel.socket.nio.SocketReceiveBufferAllocator.newBuffer(SocketReceiveBufferAllocator.java:64) ~[netty-3.10

最低0.47元/天解锁文章