生产服务器CPU占用率过高排查过程

一、问题详情

      现象：API接口访问耗时过长，排查发现当前节点内存使用3.9G,CPU占用率295%。

      当前节点已两周没发版，怀疑内存没有释放，可能是JVM垃圾回收的问题。

二、排查过程

    1、定位问题进程

    进入服务器，执行TOP命令：top

    

    发现pid为1820的进程占用了大量的CPU资源，CPU占用率高达776.1%，内存占用率也达到了29.8%。

    2、定位问题线程

    查看该进程的线程情况，执行命令：ps -mp pid -o THREAD,tid,time

    

    发现该进程的TID为1820的线程占用率很高

    3、查看问题线程堆栈

    挑选TID为1820的线程，查看该线程的堆栈情况，

    先将线程id转为16进制，执行转换命令：printf “%x\n” tid

    

    4、查看进程的内存情况

    执行命令：jstat -gcutil pid period（间隔时间） times​​​​​​​（打印次数）

    5、打印线程堆栈信息

    执行jstack命令：jstack pid |grep tid -A 30

    6、输出异常堆栈日志文件

    执行jstat命令：jstat pid >> jstat.out 

    7、定位日志

"http-nio-8765-exec-171" #15386 daemon prio=5 os_prio=0 tid=0x00007f931985e800 nid=0x3ba6 runnable [0x00007f92684c1000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.thoughtworks.xstream.XStream.registerConverterDynamically(XStream.java:1130)
	at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:1086)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:592)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:514)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:483)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:429)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:368)
	at com.baiwang.invoice.model.request.impl.XmlRequestParser.beanToXml(XmlRequestParser.java:68)
	at com.baiwang.invoice.model.request.impl.XmlRequestParser.parser(XmlRequestParser.java:41)
	at com.baiwang.invoice.service.impl.BwApiClientServiceImpl.callBwApi(BwApiClientServiceImpl.java:101)
	at com.baiwang.invoice.service.impl.BwApiClientServiceImpl.exucte(BwApiClientServiceImpl.java:60)
	at com.baiwang.invoice.service.impl.BwApiClientServiceImpl$$FastClassBySpringCGLIB$$284a950b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.baiwang.invoice.service.impl.BwApiClientServiceImpl$$EnhancerBySpringCGLIB$$84f15f30.exucte(<generated>)
	at com.baiwang.invoice.controller.BwInvoiceController.uploadInvoice(BwInvoiceController.java:67)
	at com.baiwang.invoice.controller.BwInvoiceController$$FastClassBySpringCGLIB$$c03fb451.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:52)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.baiwang.invoice.controller.BwInvoiceController$$EnhancerBySpringCGLIB$$46a17bfd.uploadInvoice(<generated>)
	at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

   8、分析问题

    可以看出是XStream类导致内存泄露。

    排查代码：

    

    原因：

    每个请求都会new一个XStream对象，然后xstream内部又会new一个CompositeClassLoader，并且Class.forName调用该loader，minor gc不会回收这种class loader对象，那就会导致heap被占满并full gc了。

   解决方法：

private static final XStream xstream = new XStream();

 至此，排查完毕。

转载于:https://my.oschina.net/u/925119/blog/1787058