背景
线上服务挂掉了,日志显示如下:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:719)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:494)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
打开线程一看,看到满屏的阻塞线程,这里截取一小端:
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000fffc54157800 JavaThread "pool-2-thread-31330" [_thread_blocked, id=51355, stack(0x0000ffeac1bf0000,0x0000ffeac1cf0000)]
=>0x0000fffc59e0c800 JavaThread "pool-2-thread-31329" [_thread_new, id=51354, stack(0x0000ffeac1cf0000,0x0000ffeac1df0000)]
0x0000fffb725ae000 JavaThread "pool-2-thread-31328" [_thread_blocked, id=51353, stack(0x0000ffeac1df0000,0x0000ffeac1ef0000)]
0x0000fffb82c49000 JavaThread "pool-2-thread-31327" [_thread_blocked, id=51352, stack(0x0000ffeac1ef0000,0x0000ffeac1ff0000)]
0x0000fffb8e550000 JavaThread "pool-2-thread-31326" [_thread_blocked, id=51351, stack(0x0000ffeac1ff0000,0x0000ffeac20f0000)]
0x0000fff2dc1b8000 JavaThread "pool-2-thread-31325" [_thread_blocked, id=51350, stack(0x0000ffeac20f0000,0x0000ffeac21f0000)]
0x0000fff2d41cf000 JavaThread "pool-2-thread-31324" [_thread_blocked, id=51349, stack(0x0000ffeac21f0000,0x0000ffeac22f0000)]
0x0000fff42c1a2000 JavaThread "pool-2-thread-31323" [_thread_blocked, id=51348, stack(0x0000ffeac22f0000,0x0000ffeac23f0000)]
0x0000fff5580cd000 JavaThread "pool-2-thread-31322" [_thread_blocked, id=51347, stack(0x0000ffeac23f0000,0x0000ffeac24f0000)]
0x0000fffa54164800 JavaThread "pool-2-thread-31321" [_thread_blocked, id=51346, stack(0x0000ffeac24f0000,0x0000ffeac25f0000)]
0x0000fffa6010f000 JavaThread "pool-2-thread-31320" [_thread_blocked, id=51345, stack(0x0000ffeac25f0000,0x0000ffeac26f0000)]
0x0000fffa4c143800 JavaThread "pool-2-thread-31319" [_thread_blocked, id=51344, stack(0x0000ffeac26f0000,0x0000ffeac27f0000)]
0x0000fffa04151000 JavaThread "pool-2-thread-31318" [_thread_blocked, id=51343, stack(0x0000ffeac27f0000,0x0000ffeac28f0000)]
0x0000fff38c107000 JavaThread "pool-2-thread-31317" [_thread_blocked, id=51342, stack(0x0000ffeac28f0000,0x0000ffeac29f0000)]
0x0000fff38c105000 JavaThread "pool-2-thread-31316" [_thread_blocked, id=51341, stack(0x0000ffeac29f0000,0x0000ffeac2af0000)]
0x0000fff444148000 JavaThread "pool-2-thread-31315" [_thread_blocked, id=51340, stack(0x0000ffeac2af0000,0x0000ffeac2bf0000)]
0x0000fff460140000 JavaThread "pool-2-thread-31314" [_thread_blocked, id=51339, stack(0x0000ffead4000000,0x0000ffead4100000)]
0x0000fff7ec092800 JavaThread "pool-2-thread-31313" [_thread_blocked, id=51338, stack(0x0000ffead4100000,0x0000ffead4200000)]
0x0000fff41c15f000 JavaThread "pool-2-thread-31312" [_thread_blocked, id=51331, stack(0x0000ffead4200000,0x0000ffead4300000)]
0x0000fff644153800 JavaThread "pool-2-thread-31311" [_thread_blocked, id=51330, stack(0x0000ffead4300000,0x0000ffead4400000)]
0x0000fff62813c000 JavaThread "pool-2-thread-31310" [_thread_blocked, id=51329, stack(0x0000ffead4400000,0x0000ffead4500000)]
0x0000fff4a011b000 JavaThread "pool-2-thread-31309" [_thread_blocked, id=51328, stack(0x0000ffead4500000,0x0000ffead4600000)]
0x0000fffb7dfd8000 JavaThread "pool-2-thread-31308" [_thread_blocked, id=51327, stack(0x0000ffead4600000,0x0000ffead4700000)]
0x0000fff46013e000 JavaThread "pool-2-thread-31307" [_thread_blocked, id=51326, stack(0x0000ffead4700000,0x0000ffead4800000)]
0x0000fff95c192800 JavaThread "pool-2-thread-31306" [_thread_blocked, id=51325, stack(0x0000ffead4800000,0x0000ffead4900000)]
0x0000fff96412d800 JavaThread "pool-2-thread-31305" [_thread_blocked, id=51324, stack(0x0000ffead4900000,0x0000ffead4a00000)]
0x0000fff9fc183000 JavaThread "pool-2-thread-31304" [_thread_blocked, id=51323, stack(0x0000ffead4a00000,0x0000ffead4b00000)]
0x0000fff4b0101800 JavaThread "pool-2-thread-31298" [_thread_blocked, id=51322, stack(0x0000ffead4b00000,0x0000ffead4c00000)]
0x0000fff4bc10e000 JavaThread "pool-2-thread-31303" [_thread_blocked, id=51321, stack(0x0000ffead4c00000,0x0000ffead4d00000)]
0x0000fff6a0196800 JavaThread "pool-2-thread-31302" [_thread_blocked, id=51320, stack(0x0000ffead4d00000,0x0000ffead4e00000)]
结合dump的日志,找到了阻塞的线程代码,定位出问题应该在了线程没有释放掉,线程释放线代码如下:
if (socket != null && socket.isAlive()) {
socket.close();
}
这里代码做了一些修改,socket是我自己创建的线程,当线程不为null,并且处于存活的时候,进行关闭操作。
问题就出在判断存活上。虽然我的socket继承了Thread,但是我没有直接调用它的start方法,而是通过guava的ListeningExecutorService的submit方法来执行,其实就是当runnable使用了。它的isAlive的结果一直都是false,所以并不会进入close方法。当时写完了之后还得意洋洋,认为更完善了,却不知埋下了祸端。
希望大家能以我的教训作为警戒。