dubbo超时机制导致的连接池耗尽处理

场景问题

dubbo超时机制导致的雪崩连接

异常描述

1. 中台Dubbo服务提供者(provider)出现无法获取Dubbo服务处理线程异常

2. 后台出现无法获取数据库连接池的异常

3. 前台响应时间异常飙高

4. 系统处理能力下降，核心基础服务无法提供正常服务

模拟场景

1. dubbo服务DemoService

/**
 * dubboDemoService
 */
public class DemoServiceImpl implements DemoService {

     public String sayHello(String name) {
         System.out.println("[" + newSimpleDateFormat("HH:mm:ss").format(new Date()) + "] Hello " + name + ",request from consumer: " + RpcContext.getContext().getRemoteAddress());
         return "Hello " + name + ", response from provider: " + RpcContext.getContext().getLocalAddress();
     }

 /**
 * DemoServiceprintUUID
 * @param uuid
 * @return
 */
 public String printUUID(String uuid) {
     System.out.println("[" + new SimpleDateFormat("HH:mm:ss").format(new Date()) + "] UUID:" + uuid + ",request from consumer: " + RpcContext.getContext().getRemoteAddress());
     try {
         Thread.sleep(30000); // 睡30s
     } catch (InterruptedException e) {
         e.printStackTrace();
     }
     return "UUID " + uuid + ", response from provider: " + RpcContext.getContext().getLocalAddress();
 }
}

2. 服务提供者用Spring xml配置声明暴露服务

 <beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:dubbo="http://code.alibabatech.com/schema/dubbo"
 xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
 http://code.alibabatech.com/schema/dubbo
http://code.alibabatech.com/schema/dubbo/dubbo.xsd">

 <!-- -->
 <dubbo:application name="hello-world-app" />

 <!-- multicast -->
 <dubbo:registry id="registry1" address="zookeeper://127.0.0.1:2181" />

 <!-- dubbo20880 -->
 <dubbo:protocol name="dubbo" port="20880" />

 <!-- bean -->
 <bean id="demoService"class="com.alibaba.dubbo.demo.provider.DemoServiceImpl" />
 
 <!-- -->
 <dubbo:service interface="com.alibaba.dubbo.demo.DemoService" ref="demoService" group="dubbo-demo" version="1.0.0" />
 
</beans>

3. 服务消费者通过Spring配置引用远程服务

<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:dubbo="http://code.alibabatech.com/schema/dubbo"
 xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
 http://code.alibabatech.com/schema/dubbo
http://code.alibabatech.com/schema/dubbo/dubbo.xsd">

 <!-- -->
 <dubbo:application name="consumer-of-helloworld-app" />

 <!-- multicast -->
 <dubbo:registry address="zookeeper://127.0.0.1:2181" />

 <!-- beandemoService -->
 <dubbo:reference id="demoService" interface="com.alibaba.dubbo.demo.DemoService" group="dubbo-demo" version="1.0.0" timeout="5000" />
</beans>

4. 服务消费者创建多线程调用100次dubbo服务

public class DemoAction {

     private DemoService demoService;

     public void setDemoService(DemoService demoService) {
         this.demoService = demoService;
     }

     /**
      * 100 DemoService
      * @throws Exception
      */
     public void start() throws Exception {
         for (int i = 0; i < 100; i++) {
             String uuid = java.util.UUID.randomUUID().toString();
             new Thread(new MyRunnable(demoService, uuid),
             String.valueOf(i)).start();
         }
     }
}

    /**
     * DemoService 
     */
    class MyRunnable implements Runnable {

         private DemoService demoService;

         private String uuid;

         public MyRunnable(DemoService demoService, String uuid) {
             this.demoService = demoService;
             this.uuid = uuid;
         }

         public void run() {
             try {
                 String uuid2 = demoService.printUUID(uuid);
                 System.out.println("[" + new SimpleDateFormat("HH:mm:ss").format(new Date()) + "] " + uuid2);
             } catch (Exception e) {
                 e.printStackTrace();
             }
         }
}

模拟结果

1. dubbo服务提供者报错日志分析： dubbo线程池耗尽 ；从日志中我们可以看到线程池的大小为200

问题：我们服务消费者只创建了100个线程调用dubbo服务，为什么会导致线程超过200？

2. dubbo服务消费者报错日志分析：从日志中可以看出调用执行DemoService的printUUID方法超时（timeout默认5000ms）。由于我们 的dubbo重试机制，如果调用一直超时，dubbo框架会默认再创建线程重试调用3次，所以如果没有dubbo线程池的限制以及一直调用服务超时的问题，消费者会一共调用400次，这就解答了问题1中dubbo服务提供者的线程池耗尽的异常了。

问题分析以及解决方案

方案1. 将dubbo的线程池设置为更大值，处理高并发调用服务的问题

方案2. 关闭dubbo消费端重试机制，防止服务调用超时后重试机制导致的耗费dubbo线程池的问题以及后台数据库连接池的耗费。

方案3、也可以同时给服务提供者和消费者的配置中都增加超时时间，出现问题后快速熔断，超时频发时并告警通知到研发人员