Java应用OOM问题排查案例

最新推荐文章于 2024-07-07 17:57:43 发布

为之则易不为则难

最新推荐文章于 2024-07-07 17:57:43 发布

阅读量702

点赞数 22

文章标签： java 开发语言

本文链接：https://blog.csdn.net/ynh_bolg/article/details/139483785

版权

文章目录

案例1 服务接口无响应
案例2 接口响应慢
- 现象
- 排查步骤

案例1 服务接口无响应

现象

早上9点左右，发现服务接口无响应，具体发生时间不确定

排查步骤

1.查看日志文件，包含大量的mq消息处理失败信息，查看相关代码后，发现此处使用线程池包裹，所以推测与此处无关。

2.寻找是否有oom文件输出

无输出

3.保存java进程此时dump文件
#输出dump

jmap -dump:format=b,file=/tmp/heapdump.bin <PID>

4.保存线程状态

jstack <PID> > /tmp/jstack_output.txt

5.检查进程是否在监听服务端口

lsof -i:6500

监听正常

6.调用该服务任意接口，进行tcpdump抓包，保存抓包数据，稍后分析

7.使用MAT分析dump文件,发现并无异常

8.查看jstack信息，无异常，http线程池全部空闲中

？"http-nio-6500-BlockPoller" #241 daemon prio=5 os_prio=0 tid=0x00007f047558c800 nid=0x2d685 runnable [0x00007f04238fb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x0000000082fd8288> (a sun.nio.ch.Util$3)
	- locked <0x0000000082fd8278> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000082fd8160> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:313)

9.分析抓包数据，发现有大量非正常tcp数据包，怀疑是网络问题，询问运维人员，最近是否有网络变动，回复为无变动。

10.怀疑安装了安全检测类软件，运维人员反馈未安装。

11.先解决mq异常问题，防止异常日志干扰。

12.因重点怀疑oom，所以直接在日志中搜索OutOfMemoryError，发现确实产生了oom，但为什么oom后端口依然有监听待查。虽然日志显示了什么时候出现了oom，但是并不能说明是此处导致的。所以还需要oom文件分析。

2024-01-31 03:52:24.092 {quartz_fcs_Worker-61} ERROR  com.test.common.quartz.statistics.ChannelMaintenanceDayStatisticsJob.execute - [ChannelMaintenanceDayStatisticsJob.java:55] ===>ChannelMaintenanceDayStatisticsJob error : {}
org.springframework.jdbc.UncategorizedSQLException: Error attempting to get column 'DOSSIER_ANAYLSE_ID' from result set.  Cause: java.sql.SQLException: Error
; uncategorized SQLException; SQL state [null]; error code [0]; Error; nested exception is java.sql.SQLException: Error
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:89)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
	at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:74)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
	at com.sun.proxy.$Proxy119.selectList(Unknown Source)
	at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:223)
	at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.executeForMany(MybatisMapperMethod.java:158)
	at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:76)
	at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61)
	at com.sun.proxy.$Proxy138.selectList(Unknown Source)
	at com.test.core.AbstractService.findByCondition(AbstractService.java:141)
	at com.test.business.channel.service.FcsChannelServiceImpl.byConditonCurrent(FcsChannelServiceImpl.java:413)
	at com.test.business.channel.service.FcsChannelServiceImpl.byCondition(FcsChannelServiceImpl.java:399)
	at com.test.business.channel.service.FcsChannelServiceImpl$$FastClassBySpringCGLIB$$c985ad2.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
	at com.test.business.channel.service.FcsChannelServiceImpl$$EnhancerBySpringCGLIB$$d3b1a743.byCondition(<generated>)
	at com.test.common.quartz.statistics.ChannelMaintenanceDayStatisticsJob.updateChannelCaptureTodayCount(ChannelMaintenanceDayStatisticsJob.java:62)
	at com.test.common.quartz.statistics.ChannelMaintenanceDayStatisticsJob.execute(ChannelMaintenanceDayStatisticsJob.java:53)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.sql.SQLException: Error
	at com.alibaba.druid.pool.DruidDataSource.handleConnectionException(DruidDataSource.java:1715)
	at com.alibaba.druid.pool.DruidPooledConnection.handleException(DruidPooledConnection.java:133)
	at com.alibaba.druid.pool.DruidPooledStatement.checkException(DruidPooledStatement.java:82)
	at com.alibaba.druid.pool.DruidPooledResultSet.checkException(DruidPooledResultSet.java:55)
	at com.alibaba.druid.pool.DruidPooledResultSet.getString(DruidPooledResultSet.java:259)
	at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:37)
	at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:26)
	at org.apache.ibatis.type.BaseTypeHandler.getResult(BaseTypeHandler.java:85)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyAutomaticMappings(DefaultResultSetHandler.java:560)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:402)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:354)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:328)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:301)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:194)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
	at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:67)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
	at com.github.pagehelper.util.SqlUtil.doIntercept(SqlUtil.java:169)
	at com.github.pagehelper.util.SqlUtil.intercept(SqlUtil.java:84)
	at com.github.pagehelper.PageHelper.intercept(PageHelper.java:50)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy166.query(Unknown Source)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
	at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
	... 24 common frames omitted
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.String.toLowerCase(String.java:2590)
	at org.mariadb.jdbc.internal.com.read.dao.ColumnNameMap.getIndex(ColumnNameMap.java:84)
	at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.findColumn(SelectResultSet.java:1484)
	at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.getString(SelectResultSet.java:1017)
	at com.alibaba.druid.pool.DruidPooledResultSet.getString(DruidPooledResultSet.java:257)
	at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:37)
	at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:26)
	at org.apache.ibatis.type.BaseTypeHandler.getResult(BaseTypeHandler.java:85)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyAutomaticMappings(DefaultResultSetHandler.java:560)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:402)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:354)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:328)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:301)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:194)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
	at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:67)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
	at com.github.pagehelper.util.SqlUtil.doIntercept(SqlUtil.java:169)
	at com.github.pagehelper.util.SqlUtil.intercept(SqlUtil.java:84)
	at com.github.pagehelper.PageHelper.intercept(PageHelper.java:50)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy166.query(Unknown Source)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
	at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
	at com.sun.proxy.$Proxy119.selectList(Unknown Source)

13.查看oom文件输出配置，和指定输出目录。发现不止是没有oom文件，/log目录下连dir1/dir2文件夹都没有，怀疑是没有指定路径导致oom文件输出失败。手动创建目录，并设置为777权限。

ps -ef | grep xxxService
sysUser   577646  577644  1  2023 ?        22:23:04 java -Xmx2G -Xms1G -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/log/dir1/dir2

14.等待复现，发现这次成功输出了oom文件。

15.使用mat分析oom文件。点击leak supects（泄漏疑点）。点击后，mat帮我们分析到了一个可疑点。点击See stacktrace可以查看线程的堆栈信息。


The thread org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0x81e27d40 quartz_fcs_Worker-32 keeps local variables with total size 1,520,211,952 (91.47%) bytes.
The memory is accumulated in one instance of "org.quartz.simpl.SimpleThreadPool$WorkerThread" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader @ 0x8007cd48".
The stacktrace of this Thread is available. See stacktrace.


Keywords
org.quartz.simpl.SimpleThreadPool$WorkerThread
org.springframework.boot.loader.LaunchedURLClassLoader @ 0x8007cd48

Details »

Leak Suspects
» Leaks
» Problem Suspect 1
» Description
» Thread Stack

 Thread Stack


quartz_fcs_Worker-32
  at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I (Native Method)
  at java.net.SocketInputStream.socketRead(Ljava/io/FileDescriptor;[BIII)I (SocketInputStream.java:116)
  at java.net.SocketInputStream.read([BIII)I (SocketInputStream.java:171)
  at java.net.SocketInputStream.read([BII)I (SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(Ljava/io/InputStream;[BII)I (InputRecord.java:465)
  at sun.security.ssl.InputRecord.readV3Record(Ljava/io/InputStream;Ljava/io/OutputStream;)V (InputRecord.java:593)
  at sun.security.ssl.InputRecord.read(Ljava/io/InputStream;Ljava/io/OutputStream;)V (InputRecord.java:532)
  at sun.security.ssl.SSLSocketImpl.readRecord(Lsun/security/ssl/InputRecord;Z)V (SSLSocketImpl.java:990)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(Lsun/security/ssl/InputRecord;)V (SSLSocketImpl.java:948)
  at sun.security.ssl.AppInputStream.read([BII)I (AppInputStream.java:105)
  at org.apache.http.impl.io.SessionInputBufferImpl.streamRead([BII)I (SessionInputBufferImpl.java:137)
  at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I (SessionInputBufferImpl.java:153)
  at org.apache.http.impl.io.SessionInputBufferImpl.read([BII)I (SessionInputBufferImpl.java:205)
  at org.apache.http.impl.io.ChunkedInputStream.read([BII)I (ChunkedInputStream.java:188)
  at org.apache.http.conn.EofSensorInputStream.read([BII)I (EofSensorInputStream.java:135)
  at java.io.FilterInputStream.read([BII)I (FilterInputStream.java:133)
  at java.io.PushbackInputStream.read([BII)I (PushbackInputStream.java:186)
  at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore()Z (UTF8StreamJsonParser.java:220)
  at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed()V (UTF8StreamJsonParser.java:2401)
  at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2([CI)V (UTF8StreamJsonParser.java:2486)
  at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString()Ljava/lang/String; (UTF8StreamJsonParser.java:2466)
  at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText()Ljava/lang/String; (UTF8StreamJsonParser.java:297)
  at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:670)
  at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:878)
  at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:652)
  at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapArray(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:829)
  at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:666)
  at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:878)
  at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:652)
  at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/JavaType;)Ljava/lang/Object; (ObjectMapper.java:4526)
  at com.fasterxml.jackson.databind.ObjectMapper.readValue(Ljava/io/InputStream;Lcom/fasterxml/jackson/databind/JavaType;)Ljava/lang/Object; (ObjectMapper.java:3521)
  at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(Lcom/fasterxml/jackson/databind/JavaType;Lorg/springframework/http/HttpInputMessage;)Ljava/lang/Object; (AbstractJackson2HttpMessageConverter.java:274)
  at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(Ljava/lang/reflect/Type;Ljava/lang/Class;Lorg/springframework/http/HttpInputMessage;)Ljava/lang/Object; (AbstractJackson2HttpMessageConverter.java:243)
  at org.springframework.web.client.HttpMessageConverterExtractor.extractData(Lorg/springframework/http/client/ClientHttpResponse;)Ljava/lang/Object; (HttpMessageConverterExtractor.java:105)
  at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(Lorg/springframework/http/client/ClientHttpResponse;)Lorg/springframework/http/ResponseEntity; (RestTemplate.java:996)
  at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(Lorg/springframework/http/client/ClientHttpResponse;)Ljava/lang/Object; (RestTemplate.java:979)
  at org.springframework.web.client.RestTemplate.doExecute(Ljava/net/URI;Lorg/springframework/http/HttpMethod;Lorg/springframework/web/client/RequestCallback;Lorg/springframework/web/client/ResponseExtractor;)Ljava/lang/Object; (RestTemplate.java:739)
  at org.springframework.web.client.RestTemplate.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Lorg/springframework/web/client/RequestCallback;Lorg/springframework/web/client/ResponseExtractor;[Ljava/lang/Object;)Ljava/lang/Object; (RestTemplate.java:672)
  at org.springframework.web.client.RestTemplate.exchange(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Lorg/springframework/http/HttpEntity;Ljava/lang/Class;[Ljava/lang/Object;)Lorg/springframework/http/ResponseEntity; (RestTemplate.java:581)
  at com.test.core.AbstractClient.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Lorg/springframework/http/HttpEntity;Ljava/lang/String;)Ljava/lang/String; (AbstractClient.java:198)
  at com.test.core.AbstractClient.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Ljava/lang/String;Ljava/lang/Object;Lcom/test/common/enums/Tier;Ljava/util/Map;)Ljava/lang/String; (AbstractClient.java:190)
  at com.test.core.AbstractClient.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Ljava/lang/String;Lcom/alibaba/fastjson/JSONObject;Lcom/test/common/enums/Tier;)Ljava/lang/String; (AbstractClient.java:146)
  at com.test.client.paas.deeplearning.surveillance.SurveillanceClientImpl.list1(Ljava/lang/Integer;Ljava/lang/Integer;)Lcom/test/common/dto/ResultsDTO; (SurveillanceClientImpl.java:112)
  at com.test.business.surveillance.job.SurveillanceUpdateJob.getSurveillanceAll()Ljava/util/List; (SurveillanceUpdateJob.java:231)
  at com.test.business.surveillance.job.SurveillanceUpdateJob.execute(Lorg/quartz/JobExecutionContext;)V (SurveillanceUpdateJob.java:89)
  at org.quartz.core.JobRunShell.run()V (JobRunShell.java:202)
  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run()V (SimpleThreadPool.java:573)





Table Of Contents

Created by Eclipse Memory Analyzer

16.根据mat提示信息，查看相关代码，发现该处将某个mysql表里的数据，与调用某个服务的数据（几万甚至几十万条）全部放在了内存中。这样明显是不正确的，随着业务数据量增大，必然导致oom。

17.优化代码逻辑，问题解决。

待查问题

oom为什么不会导致进程监听正常，单无响应。

案例2 接口响应慢

现象

接口响应很慢

排查步骤

1、查看错误日志，存在oom异常


[2024-02-19 14:15:31.271][ERROR][VideoEx:http-nio-38231-exec-1][GlobalExceptionHandler:generateOldVersionResult:57][requestId=29cd4b4f-d48f09368c984c2cb70b0d35b8d45c93], 接口调用异常
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1082)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at com.test.business.uaa.route.filter.RoutingFilter.doFilter(RoutingFilter.java:115)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at com.test.base.spring.filter.LogFilter.doFilter(LogFilter.java:82)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space

2、输出dump，使用mat分析发现com.alibaba.fastjson.parser.ParserConfig @ 0x68151fb18对象占用60%以上内存

3、点击Leak Suspects查看可疑泄漏点

One instance of “com.alibaba.fastjson.parser.ParserConfig” loaded by “org.springframework.boot.loader.LaunchedURLClassLoader @ 0x680800000” occupies 3,013,204,864 (61.95%) bytes. The memory is accumulated in one instance of “com.alibaba.fastjson.util.IdentityHashMap$Entry[]” loaded by “org.springframework.boot.loader.LaunchedURLClassLoader @ 0x680800000”.

Keywords
com.alibaba.fastjson.util.IdentityHashMap$Entry[]
com.alibaba.fastjson.parser.ParserConfig
org.springframework.boot.loader.LaunchedURLClassLoader @ 0x680800000

3、

在这里插入图片描述

根据hashcode确定大对象具体是哪里。最终确定为com.alibaba.fastjson.parser.ParserConfig中的deserializers属性。

4、查询资料发现，使用fastjson的com.alibaba.fastjson.JSON#parseObject(java.lang.String, java.lang.reflect.Type, com.alibaba.fastjson.parser.Feature…)方法进行反序列化时，将会为每个Type类型缓存一个对应的ObjectDeserializer对象。我们在使用该方法时，每次都new了一个TypeToken对象，且fastjson缓存根据对象的hashcode判断是否是同一类型，这样就导致，永远不会是相同类型。而使用fastjson中的TypeReference类时，会根据泛型的具体类名是否一致来判断。所以，应该使用fastjson中的TypeReference类进行反序列化，或者自己维护一个TypeToken的缓存。但是前者更好，可以避免缓存有重复，导致效率低下，浪费资源。

为之则易不为则难

关注

22
点赞
踩
18

收藏

觉得还不错? 一键收藏
0
评论
Java应用OOM问题排查案例

4、查询资料发现，使用fastjson的com.alibaba.fastjson.JSON#parseObject(java.lang.String, java.lang.reflect.Type, com.alibaba.fastjson.parser.Feature…9.分析抓包数据，发现有大量非正常tcp数据包，怀疑是网络问题，询问运维人员，最近是否有网络变动，回复为无变动。1.查看日志文件，包含大量的mq消息处理失败信息，查看相关代码后，发现此处使用线程池包裹，所以推测与此处无关。
复制链接

扫一扫