dubbo服务端线程池耗尽Server side threadpool is exhausted

目录

 

异常信息

 问题重现

解决办法

---

异常信息

在dubbo服务化时，有时会遇到这样的异常信息：

Caused by: com.alibaba.dubbo.remoting.RemotingException: Server side(192.168.202.56,20880) threadpool is exhausted ,detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-192.168.202.56:20880, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 3986 (completed: 3786), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://192.168.202.56:20880!
	at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:245) ~[dubbo-2.6.8.jar:2.6.8]
	at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:162) ~[dubbo-2.6.8.jar:2.6.8]
	at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:135) ~[dubbo-2.6.8.jar:2.6.8]
	at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:95) ~[dubbo-2.6.8.jar:2.6.8]
	... 70 more

也就是dubbo服务端线程池耗尽，用jvisualVM查看线程状态：

 问题重现

在服务端写测试代码如下：

 @Override
    public HospitalDeptVO getDeptById(String id) {
        
        
        while(true) {
            System.out.println("当前线程------------------"+Thread.currentThread().getName());
        }
        

    }

调用端代码：

/**    
* <p><b>Description:</b> </p>  
* <p>Copyright: Copyright (c) </p>  
* @author wangzhj
* @date 2020年8月14日  
* @version 1.0  
*/  
 package com.yky.file.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationListener;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.stereotype.Component;

import com.yky.commons.tools.log.Logger;
import com.yky.commons.tools.log.LoggerFactory;
import com.yky.mdm.api.provider.basics.DeptProvider;

/**   
* <p><b>Description:</b>启动事件处理器 </p>  
* @author wangzhj 
* @date 2020年8月14日  
*/
@Component
public class ProcessorBoot implements ApplicationListener<ContextRefreshedEvent> {


    @Autowired
    DeptProvider deptProvider;
    
    /** 
     * <p><b>Description:</b> </p>  
     * @param arg0  
     * @see org.springframework.context.ApplicationListener#onApplicationEvent(org.springframework.context.ApplicationEvent)  
     */
    @Override
    public void onApplicationEvent(ContextRefreshedEvent event) {
        if(event.getApplicationContext().getParent() == null){
            
            for(int i = 0;;i++) {
                try {
                    deptProvider.getDeptById("xxx");
                } catch (Exception e) {
                    // TODO Auto-generated catch block
                     e.printStackTrace();
                }
                System.out.println("第"+i+"次调用");
            }
//            while(true) {
//                deptProvider.getDeptById("xxx");
//                //Thread.sleep(1000);
//       
//            }
            
        }
         
    }

}

问题的原因是服务提供端的线程一直不释放（while true循环），最终导致服务端线程耗尽。

解决办法

在服务端配置executes参数。

如果是以上代码的逻辑，调用端为单线程，不存在多线程并发调用时，在调用端配置actives参数是不起作用的，原因看以下代码ActiveLimitFilter：

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.alibaba.dubbo.rpc.filter;

import com.alibaba.dubbo.common.Constants;
import com.alibaba.dubbo.common.URL;
import com.alibaba.dubbo.common.extension.Activate;
import com.alibaba.dubbo.rpc.Filter;
import com.alibaba.dubbo.rpc.Invocation;
import com.alibaba.dubbo.rpc.Invoker;
import com.alibaba.dubbo.rpc.Result;
import com.alibaba.dubbo.rpc.RpcException;
import com.alibaba.dubbo.rpc.RpcStatus;

/**
 * LimitInvokerFilter
 */
@Activate(group = Constants.CONSUMER, value = Constants.ACTIVES_KEY)
public class ActiveLimitFilter implements Filter {

    @Override
    public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
        URL url = invoker.getUrl();
        String methodName = invocation.getMethodName();
        int max = invoker.getUrl().getMethodParameter(methodName, Constants.ACTIVES_KEY, 0);
        RpcStatus count = RpcStatus.getStatus(invoker.getUrl(), invocation.getMethodName());
        if (max > 0) {
            long timeout = invoker.getUrl().getMethodParameter(invocation.getMethodName(), Constants.TIMEOUT_KEY, 0);
            long start = System.currentTimeMillis();
            long remain = timeout;
            int active = count.getActive();
            if (active >= max) {
                synchronized (count) {
                    while ((active = count.getActive()) >= max) {
                        try {
                            count.wait(remain);
                        } catch (InterruptedException e) {
                        }
                        long elapsed = System.currentTimeMillis() - start;
                        remain = timeout - elapsed;
                        if (remain <= 0) {
                            throw new RpcException("Waiting concurrent invoke timeout in client-side for service:  "
                                    + invoker.getInterface().getName() + ", method: "
                                    + invocation.getMethodName() + ", elapsed: " + elapsed
                                    + ", timeout: " + timeout + ". concurrent invokes: " + active
                                    + ". max concurrent invoke limit: " + max);
                        }
                    }
                }
            }
        }
        try {
            long begin = System.currentTimeMillis();
            //调用前并发数加1,单线程时，此时加1后变为1
            RpcStatus.beginCount(url, methodName);
            try {
                 //调用服务
                Result result = invoker.invoke(invocation);
                //调用服务后并发数减1,单线程时，此时加1后变为0
                RpcStatus.endCount(url, methodName, System.currentTimeMillis() - begin, true);
                return result;
            } catch (RuntimeException t) {
                //调用服务后并发数减1,单线程时，此时加1后变为0
                RpcStatus.endCount(url, methodName, System.currentTimeMillis() - begin, false);
                throw t;
            }
        } finally {
            if (max > 0) {
                synchronized (count) {
                    count.notify();
                }
            }
        }
    }

}

以上代码有如下的逻辑：
           

             //调用前并发数加1
            RpcStatus.beginCount(url, methodName);
            try {

                //调用服务
                Result result = invoker.invoke(invocation);

                //调用后并发数减1
                RpcStatus.endCount(url, methodName, System.currentTimeMillis() - begin, true);
                return result;
            } catch (RuntimeException t) {

               //调用后并发数减1
                RpcStatus.endCount(url, methodName, System.currentTimeMillis() - begin, false);
                throw t;
            }

因为是单线程，最大并发数始终是1，用于不会超过在调用端设置的actives参数，所以但线程情况下调用端配置actives是不起作用的。