因为想当然,服务被我搞挂了

背景
线上服务挂掉了,日志显示如下:

java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:719)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
	at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:494)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)

打开线程一看,看到满屏的阻塞线程,这里截取一小端:

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x0000fffc54157800 JavaThread "pool-2-thread-31330" [_thread_blocked, id=51355, stack(0x0000ffeac1bf0000,0x0000ffeac1cf0000)]
=>0x0000fffc59e0c800 JavaThread "pool-2-thread-31329" [_thread_new, id=51354, stack(0x0000ffeac1cf0000,0x0000ffeac1df0000)]
  0x0000fffb725ae000 JavaThread "pool-2-thread-31328" [_thread_blocked, id=51353, stack(0x0000ffeac1df0000,0x0000ffeac1ef0000)]
  0x0000fffb82c49000 JavaThread "pool-2-thread-31327" [_thread_blocked, id=51352, stack(0x0000ffeac1ef0000,0x0000ffeac1ff0000)]
  0x0000fffb8e550000 JavaThread "pool-2-thread-31326" [_thread_blocked, id=51351, stack(0x0000ffeac1ff0000,0x0000ffeac20f0000)]
  0x0000fff2dc1b8000 JavaThread "pool-2-thread-31325" [_thread_blocked, id=51350, stack(0x0000ffeac20f0000,0x0000ffeac21f0000)]
  0x0000fff2d41cf000 JavaThread "pool-2-thread-31324" [_thread_blocked, id=51349, stack(0x0000ffeac21f0000,0x0000ffeac22f0000)]
  0x0000fff42c1a2000 JavaThread "pool-2-thread-31323" [_thread_blocked, id=51348, stack(0x0000ffeac22f0000,0x0000ffeac23f0000)]
  0x0000fff5580cd000 JavaThread "pool-2-thread-31322" [_thread_blocked, id=51347, stack(0x0000ffeac23f0000,0x0000ffeac24f0000)]
  0x0000fffa54164800 JavaThread "pool-2-thread-31321" [_thread_blocked, id=51346, stack(0x0000ffeac24f0000,0x0000ffeac25f0000)]
  0x0000fffa6010f000 JavaThread "pool-2-thread-31320" [_thread_blocked, id=51345, stack(0x0000ffeac25f0000,0x0000ffeac26f0000)]
  0x0000fffa4c143800 JavaThread "pool-2-thread-31319" [_thread_blocked, id=51344, stack(0x0000ffeac26f0000,0x0000ffeac27f0000)]
  0x0000fffa04151000 JavaThread "pool-2-thread-31318" [_thread_blocked, id=51343, stack(0x0000ffeac27f0000,0x0000ffeac28f0000)]
  0x0000fff38c107000 JavaThread "pool-2-thread-31317" [_thread_blocked, id=51342, stack(0x0000ffeac28f0000,0x0000ffeac29f0000)]
  0x0000fff38c105000 JavaThread "pool-2-thread-31316" [_thread_blocked, id=51341, stack(0x0000ffeac29f0000,0x0000ffeac2af0000)]
  0x0000fff444148000 JavaThread "pool-2-thread-31315" [_thread_blocked, id=51340, stack(0x0000ffeac2af0000,0x0000ffeac2bf0000)]
  0x0000fff460140000 JavaThread "pool-2-thread-31314" [_thread_blocked, id=51339, stack(0x0000ffead4000000,0x0000ffead4100000)]
  0x0000fff7ec092800 JavaThread "pool-2-thread-31313" [_thread_blocked, id=51338, stack(0x0000ffead4100000,0x0000ffead4200000)]
  0x0000fff41c15f000 JavaThread "pool-2-thread-31312" [_thread_blocked, id=51331, stack(0x0000ffead4200000,0x0000ffead4300000)]
  0x0000fff644153800 JavaThread "pool-2-thread-31311" [_thread_blocked, id=51330, stack(0x0000ffead4300000,0x0000ffead4400000)]
  0x0000fff62813c000 JavaThread "pool-2-thread-31310" [_thread_blocked, id=51329, stack(0x0000ffead4400000,0x0000ffead4500000)]
  0x0000fff4a011b000 JavaThread "pool-2-thread-31309" [_thread_blocked, id=51328, stack(0x0000ffead4500000,0x0000ffead4600000)]
  0x0000fffb7dfd8000 JavaThread "pool-2-thread-31308" [_thread_blocked, id=51327, stack(0x0000ffead4600000,0x0000ffead4700000)]
  0x0000fff46013e000 JavaThread "pool-2-thread-31307" [_thread_blocked, id=51326, stack(0x0000ffead4700000,0x0000ffead4800000)]
  0x0000fff95c192800 JavaThread "pool-2-thread-31306" [_thread_blocked, id=51325, stack(0x0000ffead4800000,0x0000ffead4900000)]
  0x0000fff96412d800 JavaThread "pool-2-thread-31305" [_thread_blocked, id=51324, stack(0x0000ffead4900000,0x0000ffead4a00000)]
  0x0000fff9fc183000 JavaThread "pool-2-thread-31304" [_thread_blocked, id=51323, stack(0x0000ffead4a00000,0x0000ffead4b00000)]
  0x0000fff4b0101800 JavaThread "pool-2-thread-31298" [_thread_blocked, id=51322, stack(0x0000ffead4b00000,0x0000ffead4c00000)]
  0x0000fff4bc10e000 JavaThread "pool-2-thread-31303" [_thread_blocked, id=51321, stack(0x0000ffead4c00000,0x0000ffead4d00000)]
  0x0000fff6a0196800 JavaThread "pool-2-thread-31302" [_thread_blocked, id=51320, stack(0x0000ffead4d00000,0x0000ffead4e00000)]

结合dump的日志,找到了阻塞的线程代码,定位出问题应该在了线程没有释放掉,线程释放线代码如下:

      if (socket != null && socket.isAlive()) {
        socket.close();
      }

这里代码做了一些修改,socket是我自己创建的线程,当线程不为null,并且处于存活的时候,进行关闭操作。

问题就出在判断存活上。虽然我的socket继承了Thread,但是我没有直接调用它的start方法,而是通过guava的ListeningExecutorService的submit方法来执行,其实就是当runnable使用了。它的isAlive的结果一直都是false,所以并不会进入close方法。当时写完了之后还得意洋洋,认为更完善了,却不知埋下了祸端。

希望大家能以我的教训作为警戒。

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值