记一次应用配置的数据库连接被打满问题

线上应用，配置的数据库连接数为50，正常情况是已经够用了，但是有天发现50个连接全部被占满，并且长时间无法恢复，重启服务后会好一段时间。

1、问题现象

Druid获取MySQL数据库连接超时，超时时间设置的为60s

2019-09-23 15:38:24.627-|XNIO-3 task-80-|ERROR-|7fba6a1c917f42878b43f937df37e575-|10.123.18.241-|-|/api/v1/qimao/admin/distributor/add-|-|Java/1.8.0_181-|c.q.k.q.web.controller.admin.DistributorController-|method:addDistributor
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 10, maxActive 10, creating 0
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
        at com.qiyi.kpp.qimao.web.service.impl.DistributorServiceImpl$$EnhancerBySpringCGLIB$$e95f0879.addDistributor(<generated>)
        at com.qiyi.kpp.qimao.web.controller.admin.DistributorController.addDistributor(DistributorController.java:174)
        at com.qiyi.kpp.qimao.web.controller.admin.DistributorController