Dolphin scheduler Druid 连接失败问题解决

Dolphin scheduler Druid 连接失败问题解决

问题现象

搭建了dolphinscheduler分布式服务，整体web系统除调度任务外运行正常，无法配置工作流，发现master日志中显示，通过druid连接mysql失败，搜索各种资料未果

dolphinscheduler-master.2020-05-20_11.0.log

// An highlighted block
### The error may exist in org/apache/dolphinscheduler/dao/mapper/CommandMapper.xml
### The error may involve org.apache.dolphinscheduler.dao.mapper.CommandMapper.getOneToRun
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com
.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 50, creating 0, createErrorCount 2957
        at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:78)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
        at com.sun.proxy.$Proxy85.selectOne(Unknown Source)
        at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159)
        at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89)
        at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61)
        at com.sun.proxy.$Proxy93.getOneToRun(Unknown Source)
        at org.apache.dolphinscheduler.dao.ProcessDao.findOneCommand(ProcessDao.java:196)
        at org.apache.dolphinscheduler.dao.ProcessDao$$FastClassBySpringCGLIB$$ee5e71ba.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
        at org.apache.dolphinscheduler.dao.ProcessDao$$EnhancerBySpringCGLIB$$f76f5ed6.findOneCommand(<generated>)
        at org.apache.dolphinscheduler.server.master.runner.MasterSchedulerThread.run(MasterSchedulerThread.java:112)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.PersistenceException:

	### The error may involve org.apache.dolphinscheduler.dao.mapper.CommandMapper.getOneToRun
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com
.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 50, creating 0, createErrorCount 2957
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
        ... 14 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is com
.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 50, creating 0, createErrorCount 2957
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81)
        at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
        at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:67)
        at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
        at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.prepareStatement(MybatisSimpleExecutor.java:93)
        at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:66)
        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
        Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 50, creating 0, createE
rrorCount 2957
        at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1599)
        at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1326)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1306)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1296)
        at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:109)
        at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:157)
        at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115)
        at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78)
        ... 30 common frames omitted
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server
.
        at sun.reflect.GeneratedConstructorAccessor55.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown Source)

问题排查分析

检查application-dao.properties 中的各项配置

1.  重点关注了spring.datasource.url，后来通过命令行确认URL并无问题
2.  检查maxWait等参数，调整后也无用

重启 DS服务

多次重启服务，问题依旧，并未出现网上所说了开始可以连接上，后来无法连接情况

检查mysql配置

1. 增加最大连接数
2. 调整超时时间
均未能解决问题

jstack debug

 debug了一会儿，无法获取有效信息，放弃了

寻求他人帮助

找来熟悉druid的同事一起排查
1. 从application-dao.properties入手各种调整参数
2. 手动重启程序，发现程序启动时间还是之前的
 ** 找到问题了，重启脚本有问题，并未杀死原有进程，导致新配置一直未加载

总结

适当的时候请求外部人员帮助是很有必要的，换种思路，更容易找到突破口