报错信息:
2021-03-15 16:24:30.333 [http-nio-18060-exec-3706] ERROR [xxx,prod,192.168.0.172] [181527] com.xxx.xxx.common.log.LogAspect : aop发生错误 {}
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:586)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:66)
at com.xxx.xxx.svc.async.AsyncSaveSaleChance2EsGuava.async2Es(AsyncSaveSaleChance2EsGuava.java:52)
at com.xxx.xxx.svc.service.salechace.v2.service.SaleChanceEsService.saveSaleChance2Es(SaleChanceEsService.java:80)
at com.xxx.xxx.svc.service.comunication.CommunicationService.updateEsData(CommunicationService.java:174)
at com.xxx.xxx.svc.service.comunication.CommunicationService.saveCommunicationV2(CommunicationService.java:152)
at com.xxx.xxx.svc.service.comunication.CommunicationService$$FastClassBySpringCGLIB$$9ab19750.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at com.baomidou.dynamic.datasource.aop.DynamicDataSourceAnnotationInterceptor.invoke(DynamicDataSourceAnnotationInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.xxx.xxx.svc.service.comunication.CommunicationService$$EnhancerBySpringCGLIB$$e0cb33c3.saveCommunicationV2(<generated>)
at com.xxx.xxx.svc.service.tempsalechance.CrmTempSaleChanceService.saveCommunication4SaleChance(CrmTempSaleChanceService.java:212)
at com.xxx.xxx.svc.service.tempsalechance.CrmTempSaleChanceService.callBlindly(CrmTempSaleChanceService.java:180)
at com.xxx.xxx.svc.service.tempsalechance.CrmTempSaleChanceService$$FastClassBySpringCGLIB$$50f842ab.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at com.baomidou.dynamic.datasource.aop.DynamicDataSourceAnnotationInterceptor.invoke(DynamicDataSourceAnnotationInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.xxx.xxx.svc.service.tempsalechance.CrmTempSaleChanceService$$EnhancerBySpringCGLIB$$fbeba6b4.callBlindly(<generated>)
at com.xxx.xxx.web.tempsalechance.controller.CrmTempSaleChanceController.coldcall(CrmTempSaleChanceController.java:81)
at com.xxx.xxx.web.tempsalechance.controller.CrmTempSaleChanceController$$FastClassBySpringCGLIB$$9e136267.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at com.xxx.xxx.common.log.LogAspect.around(LogAspect.java:58)
at sun.reflect.GeneratedMethodAccessor357.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.xxx.xxx.web.tempsalechance.controller.CrmTempSaleChanceController$$EnhancerBySpringCGLIB$$71cb7ea4.coldcall(<generated>)
at sun.reflect.GeneratedMethodAccessor1666.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.xxx.xxx.web.security.component.DynamicSecurityFilter.doFilter(DynamicSecurityFilter.java:65)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:113)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.xxx.xxx.common.config.LogRequestFilter.doFilterInternal(LogRequestFilter.java:38)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at com.xxx.xxx.web.security.component.DynamicSecurityFilter.doFilter(DynamicSecurityFilter.java:65)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:158)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at com.xxx.xxx.web.security.component.JwtAuthenticationTokenFilter.doFilterInternal(JwtAuthenticationTokenFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
过程分析
定位业务代码:
at com.xxx.xxx.svc.async.AsyncSave2EsGuava.async2Es(AsyncSave2EsGuava.java:52)
查看这里需要向线程池申请一个线程。
报错信息意思是:unable to create new native thread
, 不能创建线程
不能创建线程是因为创建的太多了,没有足够的资源去创建了,首先查看服务器资源占用:
- 首先查看服务器资源限制:
// 切换到启用应用的用户:
su wh
[wh@crm-server lib]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30631
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
open files (-n) 65535
是系统最多打开文件数量
max user processes (-u) 4096
是用户最多打开文件数量
// 找出进程
jps
[root@crm-server crm]# jps
19026 service-0.0.1-SNAPSHOT.jar
20149 Jps
// 查看用户当前创建的线程数:
ps hH 19026 p | wc -l
[wh@server lib]$ ps hH p 19026 | wc -l
2435
// 查看该进程下的线程数
[root@crm-server crm]# cat /proc/19026/status
Name: java
Umask: 0022
State: S (sleeping)
Tgid: 27323
Ngid: 0
Pid: 27323
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups: 0
VmPeak: 9289712 kB
VmSize: 9283536 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 2656756 kB
VmRSS: 2167656 kB
RssAnon: 2153452 kB
RssFile: 14204 kB
RssShmem: 0 kB
VmData: 9121612 kB
VmStk: 132 kB
VmExe: 4 kB
VmLib: 17372 kB
VmPTE: 4804 kB
VmSwap: 0 kB
Threads: 2435
SigQ: 0/30631
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000003
SigCgt: 2000000181005ccc
CapInh: 0000000000000000
CapPrm: 0000001fffffffff
CapEff: 0000001fffffffff
CapBnd: 0000001fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: vulnerable
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 1
nonvoluntary_ctxt_switches: 1
线程数达到了 2435 ,最多可以创建 4096, 目前已经达到了 2435,所以判断是程序代码中应该有问题
跟随代码查看:
ThreadPoolExecutor poolExecutor = GuavaAsyncUtils.getPool(threadGroupName);
ListeningExecutorService listeningExecutor = GuavaAsyncUtils.getListening4ThreadPool(poolExecutor);
// 这里是 GuavaAsyncUtils 的 getPool方法
public static ThreadPoolExecutor getPool(String groupName) {
return new CustomerThreadPoolExecutor(5, 8, 0, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(100), new CustomizableThreadFactory(groupName),
new ThreadPoolExecutor.DiscardPolicy());
}
poolExecutor 是个局部变量,每来一个线程都会创建一个,这样相当于没有用到线程池
改进方法1
将局部变量变更为类变量,大家公用同一个线程池
静态变量存储在方法区,在jdk8之后叫元数据空间,所有线程都可以访问的,这样才可以发挥线程池的作用
栈、本地方法、程序计数器是线程私有的,剩下的都是线程共有的,所有线程都可以访问。
@Service
@Slf4j
public class AsyncSaveSaleChance2EsGuava {
// 异步线程池
private static String threadGroupName = "异步同步数据";
private static ThreadPoolExecutor poolExecutor = GuavaAsyncUtils.getPool(threadGroupName);
private static ListeningExecutorService listeningExecutor = GuavaAsyncUtils.getListening4ThreadPool(poolExecutor);
...
}
原理:
最开始创建线程池的代码在方法内部,方法每调用一次,都会创建一个线程池,这样跟每来一个线程就 new Thread(Java的new Thread是调用底层在服务器真实的创建一个线程) 一个线程处理业务没区别,虽然方法执行完线程池的变量也会进行销毁,但是真实的线程已经在服务器上被创建,造成极大的资源浪费。另外,Linux终究是一个文件系统,一个线程也是一个文件,服务器会对用户可以创建的文件句柄做限制,所以,如果用户创建的线程数过多,往往会导致服务器响应卡顿,慢,甚至有时候连服务器都登陆不上。
所以这里的改进方法就是把局部变量变为类变量,这样的话,所有线程都可以公用此线程池进行作业,大大减少了线程的创建。修改代码后,查看了一下,该进程创建的线程数基本稳定了
[root@crm-server crm]# ps hH p 27323 | wc -l
140
改进方法2
增大服务器用户的文件句柄限制:
- 修改
// 修改 max open files , 短期修改:
ulimit -n 65535
# 也可用下属方式修改:
ulimit -SHn 65535
# 其中-S是soft软限制模式, -H指hard硬限制模式;
# 默认是软限制, 如果只指定-n, 那就会同时配置-SH.
// 修改 max open files 永久有效
vim /etc/security/limits.conf
# 添加下面内容
# nofile - 可以打开的最大文件数, *通配符表示对所有用户有效
* soft nofile 65535
* hard nofile 65535
修改完成后保存, 退出当前用户并重新登录(不用重启服务器), 当前修改就会生效.
- 修改
max user processes
( Linux系统为每个用户都设置了一个最大进程数, 这个特性可以让我们控制服务器上现有用户可以创建的进程数量.)
方法1:(推荐使用!!!)
修改/etc/security/limits.conf文件, 在文件最后添加下述内容:
* soft nproc 131072
* hard nproc 131072
方法2:(这个可以改,但是测试过不起作用)
修改/etc/security/limits.d/90-nproc.conf文件, 在文件最后添加下述内容:
# 用户进程数的默认限制, 下面这个是对root外的其他用户限制max user processes, 要注释掉:
# * soft nproc 1024
root soft nproc 131072
修改 ulimit 参考:修改ulimit说明