问题原因
dubbo推荐也是默认的线程池方案为fix pool固定线程池大小,当请求数大于该线程池大小时,线程池没有可用线程就会出现异常:[DUBBO] Thread pool is EXHAUSTED!
dubbo 的默认线程池大小为100
dubbox(丁丁网)的默认线程池大小为200
解决方案
方案1 在dubbo provider的提供者provider.xml中的每个方法提供限流参数,使所有方法的限流总额不超过协议的线程池总数
<?xml version="1.0" encoding="UTF-8"?>
<!--
- Copyright 1999-2011 Alibaba Group.
-
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:dubbo="http://code.alibabatech.com/schema/dubbo"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
http://code.alibabatech.com/schema/dubbo http://code.alibabatech.com/schema/dubbo/dubbo.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.2.xsd">
<!-- 使用dubbo协议,服务在指定端口(配置文件中配置)暴露服务 -->
<dubbo:protocol id="dubbo" name="${dubbo.protocol.name}" port="${dubbo.protocol.port}" threads="200"/>
<!-- 创建远程服务代理,声明需要暴露的服务接口 -->
<!-- 记录投注请求流水-->
<dubbo:service interface="com.hesiyuan.test.Interface1"
protocol="dubbo" version="1.0.0" ref="Interface2Service" timeout="300" executes="99"/>
<dubbo:service interface="com.hesiyuan.test.Interface2"
protocol="dubbo" version="1.0.0" ref="Interface1Service" timeout="300"
executes="99"/>
</beans>
executes参数在Dubbo 2.5.4版本之前,并发完全生效,原因是当前线程总数量的值获取非线程安全导致:
当前线程有200大小,使用中198个线程,目前在短时间内到达5个请求,那么会导致5个请求中2个能够进入线程池获取线程,剩余3个会在一段时间后触发[DUBBO] Thread pool is EXHAUSTED!
真实案例
在做压力测试时,我们设置了线程数200,方法的总executes如上配置所示,设置为199。但是依然出现
[DUBBO] Thread pool is EXHAUSTED!
Failed to invoke method " + invocation.getMethodName() + " in provider " + url + ", cause: The service using threads greater than <dubbo:service executes=\"" + max + "\" /> limited.
duboo在2.5.4的更新中解决,该executes与当前服务总线程数的判断:
原代码中的RpcStatus count(当前运行数获取非线程安全)
@Activate(group = Constants.PROVIDER, value = Constants.EXECUTES_KEY)
public class ExecuteLimitFilter implements Filter {
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
URL url = invoker.getUrl();
String methodName = invocation.getMethodName();
int max = url.getMethodParameter(methodName, Constants.EXECUTES_KEY, 0);
if (max > 0) {
RpcStatus count = RpcStatus.getStatus(url, invocation.getMethodName());
if (count.getActive() >= max) {
throw new RpcException("Failed to invoke method " + invocation.getMethodName() + " in provider " + url + ", cause: The service using threads greater than <dubbo:service executes=\"" + max + "\" /> limited.");
}
}
long begin = System.currentTimeMillis();
boolean isException = false;
RpcStatus.beginCount(url, methodName);
try {
Result result = invoker.invoke(invocation);
return result;
} catch (Throwable t) {
isException = true;
if(t instanceof RuntimeException) {
throw (RuntimeException) t;
}
else {
throw new RpcException("unexpected exception when ExecuteLimitFilter", t);
}
}
finally {
RpcStatus.endCount(url, methodName, System.currentTimeMillis() - begin, isException);
}
}
}
更改后使用信号量的代码:
// if (count.getActive() >= max) {
/**
* http://manzhizhen.iteye.com/blog/2386408
* 通过信号量来做并发控制(即限制能使用的线程数量)
* 2017-08-21 yizhenqiang
*/
executesLimit = count.getSemaphore(max);
if(executesLimit != null && !(acquireResult = executesLimit.tryAcquire())) {
throw new RpcException("Failed to invoke method " + invocation.getMethodName() + " in provider " + url + ", cause: The service using threads greater than <dubbo:service executes=\"" + max + "\" /> limited.");
}
}
Dubbo git修复问题说明:
Fixedissues
- 不能优雅停机的问题
- MonitorFilter监控统计功能阻塞rpc流程
- 动态配置:设置指定的consumer不生效,provider端动态配置删除后仍起作用
- 路由规则解析错误,导致路由规则不生效
- async异步配置意外透传
- provider并发执行限流不准确
- 社区反馈的一些小bug修复
方案2 提高dubbo协议的线程池总量大小 及 增加服务数 使最大请求线程数不超过 服务数 * 每个服务的线程池大小*0.8
备注:因为dubbo的分流默认负载方式并非绝对均匀模式,假设1000并发请求分担到2个服务上,每个服务平均需要500线程池,但峰点会比平均多10%-30%左右,所以该服务线程池大小最少800,该波动值随着服务数量增加而逐渐趋近于平均(参考一致性hash算法 增加虚拟节点个数 )
根据一般经验,在虚机CPU个数为2U(3GHZ)时,一般建议Dubbo线程池大小开至300,并在API网关处控制该服务的最大并发数小于220,这样才不会导致流量不均衡时,某一服务接受请求数大于其Dubbo线程池大小。