springcloud服务假死

前段时间一直在排查线上服务的假死问题，首先，我们服务用的是springcloud微服务架构，注册中心是eureka，每个微服务都2个实例的，由于一个服务实例假死导致网关负载到假死的那个实例从而导致访问超时，先说说假死的服务处于什么状态吧，这个服务仍然会向注册中心发送心跳，浏览器打开http://172.18.44.120:8761/eureka/instances/cloud-service-recommender-172.18.44.128:6002查看实例状态

这个心跳时间一直有变化的，也没有什么异常信息，内存也正常，但是我出注册中心把这个服务强制下线之后，这个服务不会重新注册上去，而在com.netflix.discovery.DiscoveryClient的源码（这个是eureka服务发现的一个客户端），它里面有个renew（）方法，是向注册中心发送心跳进行续约操作的代码

 /**
     * Renew with the eureka service by making the appropriate REST call
     */
    boolean renew() {
        EurekaHttpResponse<InstanceInfo> httpResponse;
        try {
            httpResponse = eurekaTransport.registrationClient.sendHeartBeat(instanceInfo.getAppName(), instanceInfo.getId(), instanceInfo, null);
            logger.debug(PREFIX + "{} - Heartbeat status: {}", appPathIdentifier, httpResponse.getStatusCode());
            if (httpResponse.getStatusCode() == 404) {
                REREGISTER_COUNTER.increment();
                logger.info(PREFIX + "{} - Re-registering apps/{}", appPathIdentifier, instanceInfo.getAppName());
                long timestamp = instanceInfo.setIsDirtyWithTime();
                boolean success = register();
                if (success) {
                    instanceInfo.unsetIsDirty(timestamp);
                }
                return success;
            }
            return httpResponse.getStatusCode() == 200;
        } catch (Throwable e) {
            logger.error(PREFIX + "{} - was unable to send heartbeat!", appPathIdentifier, e);
            return false;
        }
    }

由上面的代码我们可以看到续约的流程是发送心跳（发送服务的appName和instanceId等）给eureka server ，而eureka server 会返回传过来的instanceId也就是这个服务实例是否在注册列表存在的信息，如果不存在会返回404，就会进去上面代码的if语句中 里面主要有个register（）的方法，这个是eureka服务注册的一个方法，也就是说如果我强制服务下线，过30秒（服务默认向eureka server 发送心跳的时间）被下线服务会调用register（）把自己重新注册回eureka server
我们开始怀疑是不是这里出了什么问题，但是没什么收获
我们又想到可能是tomcat线程池满了，查了下服务的当前线程数，卡在了200多不动了，正好符合，因为我们设置的tomcat maxThreads:200 ，那是什么原因造成线程一直不卡着不动呢，我们通过jdk的工具jstack
在服务器上运行命令jstack -l pid > xxx.log 把这个服务的线程信息dump出来分析

"http-nio-7001-exec-207" #1229 daemon prio=5 os_prio=0 tid=0x00007f3b600a2000 nid=0x4da waiting on condition [0x00007f3a9205c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d09d4120> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:210)
	at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
	at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:398)
	at ch.qos.logback.classic.Logger.info(Logger.java:583)
	at com.tgw360.service.impl.WelfareServiceImpl.autoWelfare(WelfareServiceImpl.java:199)
	at com.tgw360.service.impl.WelfareServiceImpl$$FastClassBySpringCGLIB$$82ad2726.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:669)
	at com.tgw360.service.impl.WelfareServiceImpl$$EnhancerBySpringCGLIB$$f7e7edd5.autoWelfare(<generated>)
	at com.tgw360.controller.WelfareController.autoWelfare(WelfareController.java:111)
	at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:97