解决一些历史老项目Tomcat启动问题记录

先看第一个问题：

• 现象：死活起不来，抛如下错误，其他服务器是好的，这个重装了Centos都不行。

2021-03-20 13:56:33,411 [INFO] org.elasticsearch.plugins:180 - [Mr. M] modules [], plugins [], sites []
2021-03-20 13:56:33,427 [WARN] org.springframework.web.context.support.XmlWebApplicationContext:550 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchService' defined in URL [jar:file:/home/admin/data/apache-tomcat-ebs/apache-tomcat-7.0.106/webapps/ROOT/WEB-INF/lib/cms-common-2.0.0-SNAPSHOT.jar!/cn/com/ebidding/cms/module/common/service/ElasticsearchService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [cn.com.ebidding.cms.module.common.service.ElasticsearchService]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z
2021-03-20 13:56:33,438 [INFO] com.alibaba.druid.pool.DruidDataSource:1514 - {dataSource-1} closed
2021-03-20 13:56:33,438 [INFO] org.springframework.scheduling.quartz.SchedulerFactoryBean:767 - Shutting down Quartz Scheduler
2021-03-20 13:56:33,439 [INFO] org.quartz.core.QuartzScheduler:694 - Scheduler cmsScheduler_$_NON_CLUSTERED shutting down.
2021-03-20 13:56:33,439 [INFO] org.quartz.core.QuartzScheduler:613 - Scheduler cmsScheduler_$_NON_CLUSTERED paused.
2021-03-20 13:56:33,439 [DEBUG] org.quartz.simpl.SimpleThreadPool:328 - Shutting down threadpool...
2021-03-20 13:56:33,439 [DEBUG] org.quartz.simpl.SimpleThreadPool:395 - Shutdown of threadpool complete.
2021-03-20 13:56:33,439 [INFO] org.quartz.core.QuartzScheduler:771 - Scheduler cmsScheduler_$_NON_CLUSTERED shutdown complete.
2021-03-20 13:56:33,444 [ERROR] org.springframework.web.context.ContextLoader:351 - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchService' defined in URL [jar:file:/home/admin/data/apache-tomcat-ebs/apache-tomcat-7.0.106/webapps/ROOT/WEB-INF/lib/cms-common-2.0.0-SNAPSHOT.jar!/cn/com/ebidding/cms/module/common/service/ElasticsearchService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [cn.com.ebidding.cms.module.common.service.ElasticsearchService]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.apache.log4j.Logger.isTraceEnabled()Z
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1155)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)

你的思路是啥？

1. 其实，遇到NoSuchMethodError这个问题，一般就考虑是jar包版本不兼容问题
2. 然后上lib目录下看了下，居然没有log4j 的包，其他slf4j-log4jjar 还有个对应的api也都在
3. 此时也没啥好想的，从maven官网，down下来一个兼容slf4j兼容的log4j包，丢上去
4. 信心满满的以为经过一班人，找了好几礼拜都没找到的问题，我以下就定位到了
5. 然而重启后，效果依然如此，此时有点懵逼了，Tomcat居然不加载新丢进去的jar？
6. 于是怀疑是jar丢上去有权限问题，折腾半天文件赋权，一点用没有
7. 然后在考虑将整个war弄出来，分析下可能存在class冲突的jar，一一进行反编译，对比，无果。又白折腾半天
8. 然后就准备还有一招了，换Tomcat，于是从官网down了个7.0.108版本包，有倒腾半天，问题一模一样
9. 此时真的准备放弃了……太诡异了这个
10. 就在走投无路之时，想起来jvm课上讲的一些知识，和Tomcat源码课，比如双亲委派啥的，还有印象
11. 这时，想起来Tomcat加载jar的一个顺序，我就把我的log4j jar 改名为 a-xxxx，让他名字靠前点，这样我这个jar先加载，然而也是白费力气

1. 终于，想起来可能是外部环境的问题？然后结合先前现象，想想是不是可能在java环境中存在一个jar和我的jar有冲突？
2. 本来觉得不大可能是吧，他们都说系统重装过，jdk环境也换过jdk7,等等，再说了，java home下怎么会有 log4j的jar呢？
3. 冒着一丝丝希望，来到了java_home下，寻找了半天，眼睛没找着，然后敲了个命令

1. 一声wocao，居然在jre lib ext 目录下真有一个 1.2.11的版本
2. 话不多说，弄出来反编译下，果然这个版本没有那个method

1. 松了口气，内心基本坐实了是这个问题了，赶紧重命名，把服务再重启下
2. 然后不知道是哪里操作除了问题，同事重启后，依然还是那个错，让我懵逼半天。
3. 然后不甘心的自己杀掉进程，亲手重启了下，结果居然成功启动了。
4. 同事不信，他自己说要自己再重启下看看是不是真的好了，哈哈

总结下

不知道那个朋友，在jdk里放了这个jar进去，可能都是复制拷贝解压，从来就没注意到这个问题，不是门户进行了相关jar升级，估计猴年马月也找不到。

解决一个，再赠送另外一个环境的问题

1. 有点不厚道，但没办法，还是看看吧。

2021-03-20 17:05:10,671 [INFO] org.springframework.beans.factory.xml.XmlBeanDefinitionReader:317 - Loading XML bean definitions from file [/home/admin/java/apache-tomcat-8.5.57/webapps/ROOT/WEB-INF/classes/applicationContext.xml]
2021-03-20 17:05:11,634 [INFO] cn.com.ebidding.core.util.EncryptPropertyPlaceholderConfigurer:172 - Loading properties file from class path resource [zcjb.properties]
2021-03-20 17:05:11,635 [INFO] cn.com.ebidding.core.util.EncryptPropertyPlaceholderConfigurer:172 - Loading properties file from class path resource [redis-conf.properties]
2021-03-20 17:05:11,840 [INFO] org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor:155 - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2021-03-20 17:05:11,978 [WARN] org.springframework.web.context.support.XmlWebApplicationContext:550 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cmsScheduler' defined in file [/home/admin/java/apache-tomcat-8.5.57/webapps/ROOT/WEB-INF/classes/applicationContext-cms-scheduler.xml]: Initialization of bean failed; nested exception is java.lang.IncompatibleClassChangeError: Implementing class
2021-03-20 17:05:11,980 [ERROR] org.springframework.web.context.ContextLoader:351 - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cmsScheduler' defined in file [/home/admin/java/apache-tomcat-8.5.57/webapps/ROOT/WEB-INF/classes/applicationContext-cms-scheduler.xml]: Initialization of bean failed; nested exception is java.lang.IncompatibleClassChangeError: Implementing class
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:742)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
	at cn.com.ebidding.core.web.context.EbiddingContextLoaderListener.contextInitialized(EbiddingContextLoaderListener.java:46)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4689)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5155)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1125)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1858)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IncompatibleClassChangeError: Implementing class
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2394)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:855)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1327)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613)
	at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
	at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:510)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(AutowiredAnnotationBeanPostProcessor.java:247)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineConstructorsFromBeanPostProcessors(AbstractAutowireCapableBeanFactory.java:1118)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1091)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:382)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:157)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1531)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1276)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
	... 25 more
20-Mar-2021 17:05:11.986 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
20-Mar-2021 17:05:11.989 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
20-Mar-2021 17:05:12.009 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/ROOT] has finished in [7,669] ms
20-Mar-2021 17:05:12.009 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/manager]
20-Mar-2021 17:05:12.040 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/manager] has finished in [30] ms
20-Mar-2021 17:05:12.040 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/examples]
20-Mar-2021 17:05:12.147 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/examples] has finished in [107] ms
20-Mar-2021 17:05:12.147 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/docs]
20-Mar-2021 17:05:12.156 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/docs] has finished in [9] ms
20-Mar-2021 17:05:12.157 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/host-manager]
20-Mar-2021 17:05:12.167 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/home/admin/java/apache-tomcat-8.5.57/webapps/host-manager] has finished in [10] ms
20-Mar-2021 17:05:12.171 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8800"]
20-Mar-2021 17:05:12.196 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 7890 ms

1. 法眼一看，感觉又是一个jar里的class冲突问题 java.lang.IncompatibleClassChangeError: Implementing class

2. 经过简单分析，可能与定时任务框架Quarz有关

3. 于是将lib下的Quarz相关jar low出来看下

1. 发现有这3个jar，怀疑这3个jar可能存在冲突
2. 老手段，弄出过来，反编译，对比了下，发现-2.2.2和-all-1.6.3里面有不少类相同，应该是两个时期不同的版本
3. 废话不多说，弄掉一个低版本的-all，然后重启
4. 这次顺利的很，前后分析折腾不到20分钟，就搞定了。

经验教训

java工程下的第三方依赖一定要认真维护好，jdk环境一定要保持比较纯净，有特殊情况，必须写文档说明。maven工程里的pom，不要很随意，否则，总有一天会会付出如今天这样一些惨重代价。还有，排查问题的思路要开阔些，多了解一些中间件底层原理，有时候能更快定位到问题。学以致用