Tomcat启动卡住问题排查

逍遥则子

已于 2022-07-18 16:26:46 修改

阅读量3k

点赞数

文章标签： tomcat java 开发语言

于 2022-07-18 10:11:07 首次发布

本文链接：https://blog.csdn.net/linjunjie369865393/article/details/125841248

版权

mysql作为目前常用的数据库，有些问题排查还是要考虑进来的。

##问题简述

17年左右开发的一个老系统，使用jdk1.8，tomcat7发布的一个系统。架构使用stuts+Hibernet编写的项目，前段时间因为集团安全要求加固，业务偶发造成数据库报错连接异常，需要重启应用释放连接，但是这次竟然无法启动成功了，由此造成业务系统全部停止了。。。囧~~~

tomcat日志：

catalina.out日志信息
Jul 17, 2022 1:58:49 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /usr/local/apache-tomcat/tomcat6/webapps/pda
Jul 17, 2022 1:58:49 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/local/apache-tomcat/tomcat6/webapps/pda/WEB-INF/lib/geronimo-servlet_3.0_spec-1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jul 17, 2022 1:58:49 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/local/apache-tomcat/tomcat6/webapps/pda/WEB-INF/lib/servlet-api.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jul 17, 2022 1:58:56 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://www.example.org/ReturnPolicyNo/}ReturnPolicyNoService from class com.pda.gsc.ws.cxfservice.ReturnPolicyNo
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /PolicyNo
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromWSDL
INFO: Creating Service {http://docs.oasis-open.org/ws-dd/ns/discovery/2009/01}Discovery from WSDL: classpath:/org/apache/cxf/ws/discovery/wsdl/wsdd-discovery-1.1-wsdl-os.wsdl
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be soap.udp://239.255.255.250:3702
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://docs.oasis-open.org/ws-dd/ns/discovery/2009/01}DiscoveryProxy from class org.apache.cxf.jaxws.support.DummyImpl
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://come.enn.cn/CcsToPda/}CcsToPdaService from class com.pda.gsc.ws.ccsservice.CcsToPda
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /Ccssendorder
Jul 17, 2022 1:58:57 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromClass
INFO: Creating Service {http://www.enn.cn/InsuranceReturnInfo/}InsuranceReturnInfoService from class com.pda.gsc.ws.insuranceservice.InsuranceReturnInfo
Jul 17, 2022 1:58:57 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /InsuranceReturnInfo
Jul 17, 2022 1:59:50 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jul 17, 2022 1:59:50 PM org.apache.catalina.startu

###尝试排查

1. nginx转发、网络转发

出现此类问题的时候，以为是nginx转发出现问题，未进行转发数据。随后联系了服务器组与网路组排查告知应该是应用自身问题。~~~

2. 应用排查

先验证tomcat 是否正常，将应用从tomcat中移除，进行单独启动tomcat，发现tomcat可以正常启动。(中间也试过换tomcat8单独启动也是正常)~~~~

应用一直没有做为任何修改，不应该是应用的问题。为了验证，将应用在本地启动，一切正常。之后怀疑是数据库的问题，使用Navicat工具发现也链接正常。之后又从正式环境找了一台服务器部署应用启动，发现也启动正常。

3. 端口排查

验证了应用可以链接的所有外部端口，均可以正常访问。之后联系网络组告知封闭了443与80端口，是不是此问题导致的。放开之后重启也是如此。

解决问题过程中，baidu了不少文章，做一个参考。修改之后，错误依旧。

Tomcat启动一直卡在这里，无任何报错和任何输出

解决：

找到jdk1.x.x_xx/jre/lib/security/Java.security文件，在文件中找到securerandom.source这个设置项，将其改为：

securerandom.source=file:/dev/./urandom

原因：

linux或者部分unix系统提供随机数设备是/dev/random 和/dev/urandom ，

两个有区别，urandom安全性没有random高，但random需要时间间隔生成随机数。jdk默认调用random。

然后就很简单啦，找到对应的配置文件去修改就好了

最后使用jstack 进行监控进程ID，进行排查，发现jdbc卡在RUNNABLE状态，导致有请求，无返回。但是使用Navicat可以正常链接，抱着试一试的问题，咨询DBA，告知也是正常。我就把这个错误信息发给了他。数据库排查发现 Waiting for table flush，确实有问题，因为最近在做数据库不备份，参数有问题，需要调整。之后重启应用解决。

困扰了2天的问题，终于解决。排查过程中真的是自我怀疑呀。

jstack -l 60735 >> thread.txt

thread.txt信息

"localhost-startStop-1" #14 daemon prio=5 os_prio=0 tid=0x00007fc208006000 nid=0xed5d runnable [0x00007fc21626a000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
	at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
	at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
	- locked <0x00000006c7813408> (a com.mysql.jdbc.util.ReadAheadInputStream)
	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3163)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3620)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
	- locked <0x00000006c77fbd50> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
	at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
	- locked <0x00000006c77fbd50> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.DatabaseMetaData.extractForeignKeyFromCreateTable(DatabaseMetaData.java:1416)
	at com.mysql.jdbc.DatabaseMetaData$5.forEach(DatabaseMetaData.java:3596)
	at com.mysql.jdbc.DatabaseMetaData$5.forEach(DatabaseMetaData.java:3585)
	at com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
	at com.mysql.jdbc.DatabaseMetaData.getImportedKeys(DatabaseMetaData.java:3584)
	at com.mchange.v2.c3p0.impl.NewProxyDatabaseMetaData.getImportedKeys(NewProxyDatabaseMetaData.java:3062)
	at org.hibernate.tool.hbm2ddl.TableMetadata.initForeignKeys(TableMetadata.java:162)
	at org.hibernate.tool.hbm2ddl.TableMetadata.<init>(TableMetadata.java:60)
	at org.hibernate.tool.hbm2ddl.DatabaseMetadata.getTableMetadata(DatabaseMetadata.java:114)
	at org.hibernate.cfg.Configuration.generateSchemaUpdateScript(Configuration.java:1133)
	at org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:212)
	at org.hibernate.tool.hbm2ddl.SchemaUpdate.execute(SchemaUpdate.java:178)
	at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:503)
	at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1750)
	at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1788)
	at org.springframework.orm.hibernate4.LocalSessionFactoryBuilder.buildSessionFactory(LocalSessionFactoryBuilder.java:247)
	at org.springframework.orm.hibernate4.LocalSessionFactoryBean.buildSessionFactory(LocalSessionFactoryBean.java:373)
	at org.springframework.orm.hibernate4.LocalSessionFactoryBean.afterPropertiesSet(LocalSessionFactoryBean.java:358)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
	- locked <0x00000006c6f2fe48> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:610)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
	- locked <0x00000006c6e41860> (a java.lang.Object)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:389)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:294)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4973)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5467)
	- locked <0x00000006c633a900> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	- locked <0x00000006c633a900> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1229)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1875)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

数据库备份错误信息：

29864832 root localhost  Query 53084 Waiting for table flush FLUSH /*!40101 LOCAL */ TABLES
29846815 root localhost pda Query 65697 Sending data SELECT /*!40001 SQL_NO_CACHE */ * FROM `PoolLogs`

This is Fine

逍遥则子

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
打赏
0
评论
Tomcat启动卡住问题排查

架构使用stuts+Hibernet编写的项目，前端时间因为安全要求加固，造成数据库报错连接异常，然后需要重启应用释放连接，但是再也无法启动成功了，由此造成业务系统全部停止了。最后使用jstack进行监控进程ID，进行排查，发现jdbc卡在RUNNABLE状态，导致有请求，无返回。但是使用navcat可以正常链接，抱着试一试的问题，咨询DBA，告知也是正常。先验证tomcat是否正常，将应用从tomcat中移除，进行单独启动tomcat，发现tomcat可以正常启动。困扰了2天的问题，终于解决。......
复制链接

扫一扫