案例1 服务接口无响应
现象
早上9点左右,发现服务接口无响应,具体发生时间不确定
排查步骤
1.查看日志文件,包含大量的mq消息处理失败信息,查看相关代码后,发现此处使用线程池包裹,所以推测与此处无关。
2.寻找是否有oom文件输出
无输出
3.保存java进程此时dump文件
#输出dump
jmap -dump:format=b,file=/tmp/heapdump.bin <PID>
4.保存线程状态
jstack <PID> > /tmp/jstack_output.txt
5.检查进程是否在监听服务端口
lsof -i:6500
监听正常
6.调用该服务任意接口,进行tcpdump抓包,保存抓包数据,稍后分析
7.使用MAT分析dump文件,发现并无异常
8.查看jstack信息,无异常,http线程池全部空闲中
?"http-nio-6500-BlockPoller" #241 daemon prio=5 os_prio=0 tid=0x00007f047558c800 nid=0x2d685 runnable [0x00007f04238fb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000082fd8288> (a sun.nio.ch.Util$3)
- locked <0x0000000082fd8278> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000082fd8160> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:313)
9.分析抓包数据,发现有大量非正常tcp数据包,怀疑是网络问题,询问运维人员,最近是否有网络变动,回复为无变动。
10.怀疑安装了安全检测类软件,运维人员反馈未安装。
11.先解决mq异常问题,防止异常日志干扰。
12.因重点怀疑oom,所以直接在日志中搜索OutOfMemoryError,发现确实产生了oom,但为什么oom后端口依然有监听待查。虽然日志显示了什么时候出现了oom,但是并不能说明是此处导致的。所以还需要oom文件分析。
2024-01-31 03:52:24.092 {quartz_fcs_Worker-61} ERROR com.test.common.quartz.statistics.ChannelMaintenanceDayStatisticsJob.execute - [ChannelMaintenanceDayStatisticsJob.java:55] ===>ChannelMaintenanceDayStatisticsJob error : {}
org.springframework.jdbc.UncategorizedSQLException: Error attempting to get column 'DOSSIER_ANAYLSE_ID' from result set. Cause: java.sql.SQLException: Error
; uncategorized SQLException; SQL state [null]; error code [0]; Error; nested exception is java.sql.SQLException: Error
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:89)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:74)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
at com.sun.proxy.$Proxy119.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:223)
at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.executeForMany(MybatisMapperMethod.java:158)
at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:76)
at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61)
at com.sun.proxy.$Proxy138.selectList(Unknown Source)
at com.test.core.AbstractService.findByCondition(AbstractService.java:141)
at com.test.business.channel.service.FcsChannelServiceImpl.byConditonCurrent(FcsChannelServiceImpl.java:413)
at com.test.business.channel.service.FcsChannelServiceImpl.byCondition(FcsChannelServiceImpl.java:399)
at com.test.business.channel.service.FcsChannelServiceImpl$$FastClassBySpringCGLIB$$c985ad2.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.test.business.channel.service.FcsChannelServiceImpl$$EnhancerBySpringCGLIB$$d3b1a743.byCondition(<generated>)
at com.test.common.quartz.statistics.ChannelMaintenanceDayStatisticsJob.updateChannelCaptureTodayCount(ChannelMaintenanceDayStatisticsJob.java:62)
at com.test.common.quartz.statistics.ChannelMaintenanceDayStatisticsJob.execute(ChannelMaintenanceDayStatisticsJob.java:53)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.sql.SQLException: Error
at com.alibaba.druid.pool.DruidDataSource.handleConnectionException(DruidDataSource.java:1715)
at com.alibaba.druid.pool.DruidPooledConnection.handleException(DruidPooledConnection.java:133)
at com.alibaba.druid.pool.DruidPooledStatement.checkException(DruidPooledStatement.java:82)
at com.alibaba.druid.pool.DruidPooledResultSet.checkException(DruidPooledResultSet.java:55)
at com.alibaba.druid.pool.DruidPooledResultSet.getString(DruidPooledResultSet.java:259)
at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:37)
at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:26)
at org.apache.ibatis.type.BaseTypeHandler.getResult(BaseTypeHandler.java:85)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyAutomaticMappings(DefaultResultSetHandler.java:560)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:402)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:354)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:328)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:301)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:194)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:67)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
at com.github.pagehelper.util.SqlUtil.doIntercept(SqlUtil.java:169)
at com.github.pagehelper.util.SqlUtil.intercept(SqlUtil.java:84)
at com.github.pagehelper.PageHelper.intercept(PageHelper.java:50)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
at com.sun.proxy.$Proxy166.query(Unknown Source)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
... 24 common frames omitted
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.String.toLowerCase(String.java:2590)
at org.mariadb.jdbc.internal.com.read.dao.ColumnNameMap.getIndex(ColumnNameMap.java:84)
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.findColumn(SelectResultSet.java:1484)
at org.mariadb.jdbc.internal.com.read.resultset.SelectResultSet.getString(SelectResultSet.java:1017)
at com.alibaba.druid.pool.DruidPooledResultSet.getString(DruidPooledResultSet.java:257)
at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:37)
at org.apache.ibatis.type.StringTypeHandler.getNullableResult(StringTypeHandler.java:26)
at org.apache.ibatis.type.BaseTypeHandler.getResult(BaseTypeHandler.java:85)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyAutomaticMappings(DefaultResultSetHandler.java:560)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:402)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:354)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:328)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:301)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:194)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
at com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doQuery(MybatisSimpleExecutor.java:67)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
at com.github.pagehelper.util.SqlUtil.doIntercept(SqlUtil.java:169)
at com.github.pagehelper.util.SqlUtil.intercept(SqlUtil.java:84)
at com.github.pagehelper.PageHelper.intercept(PageHelper.java:50)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
at com.sun.proxy.$Proxy166.query(Unknown Source)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
at com.sun.proxy.$Proxy119.selectList(Unknown Source)
13.查看oom文件输出配置,和指定输出目录。发现不止是没有oom文件,/log目录下连dir1/dir2文件夹都没有,怀疑是没有指定路径导致oom文件输出失败。手动创建目录,并设置为777权限。
ps -ef | grep xxxService
sysUser 577646 577644 1 2023 ? 22:23:04 java -Xmx2G -Xms1G -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/log/dir1/dir2
14.等待复现,发现这次成功输出了oom文件。
15.使用mat分析oom文件。 点击leak supects(泄漏疑点)。点击后,mat帮我们分析到了一个可疑点。点击See stacktrace可以查看线程的堆栈信息。
The thread org.quartz.simpl.SimpleThreadPool$WorkerThread @ 0x81e27d40 quartz_fcs_Worker-32 keeps local variables with total size 1,520,211,952 (91.47%) bytes.
The memory is accumulated in one instance of "org.quartz.simpl.SimpleThreadPool$WorkerThread" loaded by "org.springframework.boot.loader.LaunchedURLClassLoader @ 0x8007cd48".
The stacktrace of this Thread is available. See stacktrace.
Keywords
org.quartz.simpl.SimpleThreadPool$WorkerThread
org.springframework.boot.loader.LaunchedURLClassLoader @ 0x8007cd48
Details »
Leak Suspects
» Leaks
» Problem Suspect 1
» Description
» Thread Stack
Thread Stack
quartz_fcs_Worker-32
at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I (Native Method)
at java.net.SocketInputStream.socketRead(Ljava/io/FileDescriptor;[BIII)I (SocketInputStream.java:116)
at java.net.SocketInputStream.read([BIII)I (SocketInputStream.java:171)
at java.net.SocketInputStream.read([BII)I (SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(Ljava/io/InputStream;[BII)I (InputRecord.java:465)
at sun.security.ssl.InputRecord.readV3Record(Ljava/io/InputStream;Ljava/io/OutputStream;)V (InputRecord.java:593)
at sun.security.ssl.InputRecord.read(Ljava/io/InputStream;Ljava/io/OutputStream;)V (InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(Lsun/security/ssl/InputRecord;Z)V (SSLSocketImpl.java:990)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Lsun/security/ssl/InputRecord;)V (SSLSocketImpl.java:948)
at sun.security.ssl.AppInputStream.read([BII)I (AppInputStream.java:105)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead([BII)I (SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I (SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.read([BII)I (SessionInputBufferImpl.java:205)
at org.apache.http.impl.io.ChunkedInputStream.read([BII)I (ChunkedInputStream.java:188)
at org.apache.http.conn.EofSensorInputStream.read([BII)I (EofSensorInputStream.java:135)
at java.io.FilterInputStream.read([BII)I (FilterInputStream.java:133)
at java.io.PushbackInputStream.read([BII)I (PushbackInputStream.java:186)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore()Z (UTF8StreamJsonParser.java:220)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed()V (UTF8StreamJsonParser.java:2401)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2([CI)V (UTF8StreamJsonParser.java:2486)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString()Ljava/lang/String; (UTF8StreamJsonParser.java:2466)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText()Ljava/lang/String; (UTF8StreamJsonParser.java:297)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:670)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:878)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:652)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapArray(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:829)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:666)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:878)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; (UntypedObjectDeserializer.java:652)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/JavaType;)Ljava/lang/Object; (ObjectMapper.java:4526)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(Ljava/io/InputStream;Lcom/fasterxml/jackson/databind/JavaType;)Ljava/lang/Object; (ObjectMapper.java:3521)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(Lcom/fasterxml/jackson/databind/JavaType;Lorg/springframework/http/HttpInputMessage;)Ljava/lang/Object; (AbstractJackson2HttpMessageConverter.java:274)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(Ljava/lang/reflect/Type;Ljava/lang/Class;Lorg/springframework/http/HttpInputMessage;)Ljava/lang/Object; (AbstractJackson2HttpMessageConverter.java:243)
at org.springframework.web.client.HttpMessageConverterExtractor.extractData(Lorg/springframework/http/client/ClientHttpResponse;)Ljava/lang/Object; (HttpMessageConverterExtractor.java:105)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(Lorg/springframework/http/client/ClientHttpResponse;)Lorg/springframework/http/ResponseEntity; (RestTemplate.java:996)
at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(Lorg/springframework/http/client/ClientHttpResponse;)Ljava/lang/Object; (RestTemplate.java:979)
at org.springframework.web.client.RestTemplate.doExecute(Ljava/net/URI;Lorg/springframework/http/HttpMethod;Lorg/springframework/web/client/RequestCallback;Lorg/springframework/web/client/ResponseExtractor;)Ljava/lang/Object; (RestTemplate.java:739)
at org.springframework.web.client.RestTemplate.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Lorg/springframework/web/client/RequestCallback;Lorg/springframework/web/client/ResponseExtractor;[Ljava/lang/Object;)Ljava/lang/Object; (RestTemplate.java:672)
at org.springframework.web.client.RestTemplate.exchange(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Lorg/springframework/http/HttpEntity;Ljava/lang/Class;[Ljava/lang/Object;)Lorg/springframework/http/ResponseEntity; (RestTemplate.java:581)
at com.test.core.AbstractClient.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Lorg/springframework/http/HttpEntity;Ljava/lang/String;)Ljava/lang/String; (AbstractClient.java:198)
at com.test.core.AbstractClient.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Ljava/lang/String;Ljava/lang/Object;Lcom/test/common/enums/Tier;Ljava/util/Map;)Ljava/lang/String; (AbstractClient.java:190)
at com.test.core.AbstractClient.execute(Ljava/lang/String;Lorg/springframework/http/HttpMethod;Ljava/lang/String;Lcom/alibaba/fastjson/JSONObject;Lcom/test/common/enums/Tier;)Ljava/lang/String; (AbstractClient.java:146)
at com.test.client.paas.deeplearning.surveillance.SurveillanceClientImpl.list1(Ljava/lang/Integer;Ljava/lang/Integer;)Lcom/test/common/dto/ResultsDTO; (SurveillanceClientImpl.java:112)
at com.test.business.surveillance.job.SurveillanceUpdateJob.getSurveillanceAll()Ljava/util/List; (SurveillanceUpdateJob.java:231)
at com.test.business.surveillance.job.SurveillanceUpdateJob.execute(Lorg/quartz/JobExecutionContext;)V (SurveillanceUpdateJob.java:89)
at org.quartz.core.JobRunShell.run()V (JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run()V (SimpleThreadPool.java:573)
Table Of Contents
Created by Eclipse Memory Analyzer
16.根据mat提示信息,查看相关代码,发现该处将某个mysql表里的数据,与调用某个服务的数据(几万甚至几十万条)全部放在了内存中。这样明显是不正确的,随着业务数据量增大,必然导致oom。
17.优化代码逻辑,问题解决。
待查问题
oom为什么不会导致进程监听正常,单无响应。
案例2 接口响应慢
现象
接口响应很慢
排查步骤
1、查看错误日志,存在oom异常
[2024-02-19 14:15:31.271][ERROR][VideoEx:http-nio-38231-exec-1][GlobalExceptionHandler:generateOldVersionResult:57][requestId=29cd4b4f-d48f09368c984c2cb70b0d35b8d45c93], 接口调用异常
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1082)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.test.business.uaa.route.filter.RoutingFilter.doFilter(RoutingFilter.java:115)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.test.base.spring.filter.LogFilter.doFilter(LogFilter.java:82)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2、输出dump,使用mat分析发现com.alibaba.fastjson.parser.ParserConfig @ 0x68151fb18对象占用60%以上内存
3、点击Leak Suspects查看可疑泄漏点
One instance of “com.alibaba.fastjson.parser.ParserConfig” loaded by “org.springframework.boot.loader.LaunchedURLClassLoader @ 0x680800000” occupies 3,013,204,864 (61.95%) bytes. The memory is accumulated in one instance of “com.alibaba.fastjson.util.IdentityHashMap$Entry[]” loaded by “org.springframework.boot.loader.LaunchedURLClassLoader @ 0x680800000”.
Keywords
com.alibaba.fastjson.util.IdentityHashMap$Entry[]
com.alibaba.fastjson.parser.ParserConfig
org.springframework.boot.loader.LaunchedURLClassLoader @ 0x680800000
3、
根据hashcode确定大对象具体是哪里。最终确定为com.alibaba.fastjson.parser.ParserConfig中的deserializers属性。
4、查询资料发现,使用fastjson的com.alibaba.fastjson.JSON#parseObject(java.lang.String, java.lang.reflect.Type, com.alibaba.fastjson.parser.Feature…)方法进行反序列化时,将会为每个Type类型缓存一个对应的ObjectDeserializer对象。 我们在使用该方法时,每次都new了一个TypeToken对象,且fastjson缓存根据对象的hashcode判断是否是同一类型,这样就导致,永远不会是相同类型。而使用fastjson中的TypeReference类时,会根据泛型的具体类名是否一致来判断。所以,应该使用fastjson中的TypeReference类进行反序列化,或者自己维护一个TypeToken的缓存。但是前者更好,可以避免缓存有重复,导致效率低下,浪费资源。