记一次使用hive-jdbc+tomcat-jdbc连接(Connection)中断的处理过程

最新推荐文章于 2023-08-11 10:49:07 发布

小饭大人

最新推荐文章于 2023-08-11 10:49:07 发布

阅读量3.4k

点赞数 1

分类专栏： DB 文章标签： hive-jdbc tomcat-jdbc Connection closed

本文链接：https://blog.csdn.net/nmyphp/article/details/103683137

版权

当业务代码遍历大量Hive表信息时，遇到连接在4000+表时突然断开。通过日志分析，发现异常源于Hive-JDBC的libthrift，连接由Mybatis到Tomcat-JDBC逐级中断。问题定位为`removeAbandonedTimeout`配置，经过调整该参数，解决了6分钟后连接被自动关闭的问题。下一步计划研究为何在使用中的连接会被标记为abandoned。

摘要由CSDN通过智能技术生成

现象描述

Hive环境一个数据库，拥有表8000+，业务代码需要挨个desc {tableName}来获取表信息。当程序运行到4000+左右，开始出现获取信息失败。

查找原因

通过查看日志发现是连接已断开，具体如下：

【INFO】org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: java.sql.SQLException: Connection has already been closed.
### The error may exist in 省略/HiveCollectionMapper.xml
### The error may involve 省略.HiveCollectionMapper.getTableParameters
### The error occurred while executing a query
### SQL: desc formatted logsget.`dim_qdas_superderive_eventlist`
### Cause: java.sql.SQLException: Connection has already been closed.
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
	at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:57)
	at com.sun.proxy.$Proxy365.getTableParameters(Unknown Source)
	...省略
		at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Connection has already been closed.
	at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:117)
	at org.apache.tomcat.jdbc.pool.JdbcIn