APPARENT DEADLOCK Creating emergency threads for unassigned pending tasks

项目场景：

在迁移CDH集群的元数据库后，重新启动集群，发现启动失败。

APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

---

问题描述

在启动cloudera-scm-server 时，日志如下

2022-08-25 17:24:22,222 INFO main:com.cloudera.server.cmf.Main: ================================================================================
2022-08-25 17:24:22,230 INFO main:com.cloudera.server.cmf.Main: Starting SCM Server. JVM Args: [-Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties, -Dfile.encoding=UTF-8, -Dcmf.root.logger=INFO,LOGFILE, -Dcmf.log.dir=/var/log/cloudera-scm-server, -Dcmf.log.file=cloudera-scm-server.log, -Dcmf.jetty.threshhold=WARN, -Dcmf.schema.dir=/opt/cloudera/cm/schema, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Dpython.home=/opt/cloudera/cm/python, -XX:+UseConcMarkSweepGC, -XX:+UseParNewGC, -XX:+HeapDumpOnOutOfMemoryError, -Xmx2G, -XX:MaxPermSize=256m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/tmp, -XX:OnOutOfMemoryError=kill -9 %p], Args: [], Version: 6.2.1 (#1426065 built by jenkins on 20190911-0956 git: 94625ae139b266e8cdad073d744766545f7f0472)
2022-08-25 17:24:22,373 INFO main:org.springframework.context.support.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@11758f2a: startup date [Thu Aug 25 17:24:22 CST 2022]; root of context hierarchy
2022-08-25 17:24:22,432 INFO main:org.springframework.beans.factory.xml.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [webapp/WEB-INF/spring/beanRefFactory.xml]
2022-08-25 17:24:22,808 INFO main:org.springframework.context.support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@671a5887: startup date [Thu Aug 25 17:24:22 CST 2022]; root of context hierarchy
2022-08-25 17:24:22,880 INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.properties
2022-08-25 17:24:22,883 INFO main:com.cloudera.enterprise.CommonMain: Statistics not enabled, c3p0 JMX disabled
2022-08-25 17:24:24,702 INFO main:com.mchange.v2.log.MLog: MLog clients using log4j logging.
2022-08-25 17:24:24,762 INFO main:com.mchange.v2.c3p0.C3P0Registry: Initializing c3p0-0.9.2.1 [built 20-March-2013 10:47:27 +0000; debug? true; trace: 10]
2022-08-25 17:24:24,852 INFO main:com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@3d216406 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@b45fca09 [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1bqvrwraq1mwrx23bgnttf|f288c14, idleConnectionTestPeriod -> 300, initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@d045e305 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 1bqvrwraq1mwrx23bgnttf|4d654825, jdbcUrl -> jdbc:mysql://172.21.85.147:60001/yptdevscm?useUnicode=true&characterEncoding=UTF-8, properties -> {user=******, password=******} ], preferredTestQuery -> null, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 1bqvrwraq1mwrx23bgnttf|53b8afea, numHelperThreads -> 3 ]
2022-08-25 17:24:44,852 WARN C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-AdminTaskTimer:com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@125fa98c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2022-08-25 17:24:44,854 WARN C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-AdminTaskTimer:com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@125fa98c -- APPARENT DEADLOCK!!! Complete Status:
        Managed Threads: 3
        Active Threads: 3
        Active Tasks:
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5da54f8c
                        on thread: C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-HelperThread-#2
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@32d99e7d
                        on thread: C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-HelperThread-#1
        Pending Tasks:
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4a245f24
                com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3d8ab0c3
Pool thread stack traces:
        Thread[C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-HelperThread-#2,5,main]
                java.net.PlainSocketImpl.socketConnect(Native Method)
                java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
                java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
                java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
                java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
                java.net.Socket.connect(Socket.java:589)
                java.net.Socket.connect(Socket.java:538)
                java.net.Socket.<init>(Socket.java:434)
                java.net.Socket.<init>(Socket.java:244)
                com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:258)
                com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:306)
                com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2504)
                com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2541)
                com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2323)
                com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
                com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
                sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
                com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
                com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
                com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
                com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
                com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
                com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
                com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)

---

原因分析：

盯着 APPARENT DEADLOCK!!! 的报错找了好久，差点要改C3P0的jar包了，

最后猛然发现，是当前机器和远程数据库的网络没有打通。。cloudera-scm-server 的日志里面也体现不出来网络不通的信息。。

迁移数据库后，环境并没有检查很仔细

---

解决方案：

环境问题真的很让人头疼