压测问题小记之OutOfMemoryError

最新推荐文章于 2024-05-28 11:04:37 发布

置顶 Donglei603790516

最新推荐文章于 2024-05-28 11:04:37 发布

阅读量1k

点赞数 3

分类专栏：性能测试文章标签：性能测试

本文链接：https://blog.csdn.net/Donglei603790516/article/details/79803240

版权

性能测试专栏收录该内容

2 篇文章 0 订阅

订阅专栏

近日，在做后台接口压测过程中，发现一个高并发下会触发的bug，bug本身并不复杂，但具有一定的代表性，特此记录一下。

接口概述

首先要测试的接口是场景险的投保接口，该接口的业务场景是考拉对接网易保险进行商品的质保或延保险投保，网易保险对接保险公司进行投保。

该接口的内部处理逻辑，如下流程示意图所示。

问题描述

本文使用jmeter对投保接口进行压力测试，增大压力开始测试后，出现很多错误请求返回，观察后台接口日志，具体错误如下：

2018-03-30 16:59:46.012 [http-nio-7112-exec-6] ERROR o.a.c.c.C.[.[localhost].[/].[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
	at com.netease.baoxian.service.InsuranceService.insured(InsuranceService.java:110)
	at com.netease.baoxian.service.ServiceOrderService.addServiceOrder(ServiceOrderService.java:126)
	at com.netease.baoxian.controller.SceneController.insure(SceneController.java:180)
	at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:186)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

原因分析

JVM向操作系统申请创建新的 native thread(原生线程)时，就有可能会碰到 java.lang.OutOfMemoryError: Unable to create new native thread 错误。如果底层操作系统创建新的 native thread 失败，JVM就会抛出相应的OutOfMemoryError。总体来说，导致 java.lang.OutOfMemoryError: Unable to create new native thread 错误的场景大多经历以下这些阶段：

Java程序向JVM请求创建一个新的Java线程；
JVM本地代码(native code)代理该请求, 尝试创建一个操作系统级别的 native thread(原生线程)；
操作系统尝试创建一个新的native thread, 需要同时分配一些内存给该线程；
如果操作系统的虚拟内存已耗尽, 或者是受到32位进程的地址空间限制(约2-4GB), OS就会拒绝本地内存分配；
JVM抛出 java.lang.OutOfMemoryError: Unable to create new native thread 错误。

问题定位

根据堆栈信息，我们定位到问题代码出现在下面这个方法里，此方法实现的功能是异步调用保险公司投保接口。

    public void insured(ServiceOrder serviceOrder, boolean isSync){
        if(isSync){
            insured(serviceOrder);
        }else{
            ExecutorService es = Executors.newFixedThreadPool(1);
            es.submit(()->insured(serviceOrder));
            es.shutdown();
        }
    }

分析这段代码，开发在处理异步任务时，使用了线程池。使用线程池处理多线程的好处有以下几点：

重用存在的线程，减少对象创建、消亡的开销，性能佳。
可有效控制最大并发线程数，提高系统资源的使用率，同时避免过多资源竞争，避免堵塞。
提供定时执行、定期执行、单线程、并发数控制等功能。

问题就出现在ExecutorService es = Executors.newFixedThreadPool(1);这一行代码上。newFixedThreadPool方法的功能是创建一个定长线程池，可控制线程最大并发数，超出的线程会在队列中等待。开发同学在此处创建线程池时使用了局部变量，每个请求进入这个方法时，都会新建一个固定线程个数为1的线程池。就这样大量并发请求进入后，新建了大量的线程，导致系统虚拟内存被耗尽，JVM抛出 java.lang.OutOfMemoryError: Unable to create new native thread 错误。

解决方案

问题定位到了，解决起来也就很简单了，只需正确使用线程池就好了。

    private static ExecutorService es = Executors.newFixedThreadPool(50);
    public void insured(ServiceOrder serviceOrder, boolean isSync){
        if(isSync){
            insured(serviceOrder);
        }else{
            es.submit(()->insured(serviceOrder));
        }
    }

开发同学修改代码如上，把线程池定义为静态成员变量，定长50个线程。这样大量请求进入此方法后，会使用这50个线程循环处理请求，超出的线程会在队列中等待。

问题已经得到解决，代码修改后，接口在高并发情况下已经不再报OOM的错误。但是参考阿里编码规约，此种线程池的使用方式仍有一定的问题。使用newFixedThreadPool方法的主要问题是，堆积的请求处理队列可能会耗费非常大的内存，甚至OOM。根据阿里编码规约建议，线程池不允许使用Executors去创建，而是通过ThreadPoolExecutor的方式，这样的处理方式能让写的同学更加明确线程池的运行规则，规避资源耗尽的风险。

    private static ThreadFactory namedThreadFactory = new ThreadFactoryBuilder()
            .setNameFormat("demo-pool-%d")
            .build();
    private static ExecutorService pool = new ThreadPoolExecutor(5, 200, 0L, TimeUnit.MILLISECONDS,
            new LinkedBlockingDeque<Runnable>(1024), namedThreadFactory, new ThreadPoolExecutor.AbortPolicy());
            
    public void insured(ServiceOrder serviceOrder, boolean isSync) {
        if (isSync) {
            insured(serviceOrder);
        } else {
            pool.execute(() -> insured(serviceOrder));
            pool.shutdown();
        }
    }

小结

至此，该问题基本得到解决。问题不大，更重要的意义在于发现并解决此问题的过程。作为测试人员，不仅要对待测系统的后台架构和实现逻辑有清晰的理解，还要掌握基本的代码技能，才能不断提高自己分析和解决问题的能力。

Donglei603790516

关注

3
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
压测问题小记之OutOfMemoryError

近日，在做后台接口压测过程中，发现一个高并发下会触发的bug，bug本身并不复杂，但具有一定的代表性，特此记录一下。接口概述首先要测试的接口是场景险的投保接口，该接口的业务场景是考拉对接网易保险进行商品的质保或延保险投保，网易保险对接保险公司进行投保。该接口的内部处理逻辑，如下流程示意图所示。问题描述本文使用jmeter对投保接口进行压力测试，增大压力开始测试后，出现很多错误请求返...
复制链接

扫一扫