在生产上,可能会遇到CPU中占用过高的问题,可能是程序陷入了死循环,假设在spring boot工程中有下面一段问题代码,那该如何通过一些简单的命令排查问题呢?
package com.wjq.demo.controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class TestController {
@RequestMapping("/test")
public void test(){
int i = 0;
while (true){
i++;
}
}
}
第一步,通过top命令查看CPU占用最高的进程
top命令类似于windows系统的资源管理器,可查看系统的资源使用情况,直接输入
top
得到
top - 20:43:13 up 2:51, 1 user, load average: 0.54, 0.16, 0.09
Tasks: 173 total, 1 running, 172 sleeping, 0 stopped, 0 zombie
%Cpu(s): 50.2 us, 0.2 sy, 0.0 ni, 49.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1863252 total, 661364 free, 659152 used, 542736 buff/cache
KiB Swap: 2097148 total, 2097148 free, 0 used. 966724 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7933 root 20 0 2990552 274124 13576 S 99.3 14.7 1:40.68 java
10169 root 20 0 162140 2424 1616 R 0.3 0.1 0:00.20 top
1 root 20 0 193804 6884 4180 S 0.0 0.4 0:02.43 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.13 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:00.16 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.86 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.09 watchdog/0
12 root rt 0 0 0 0 S 0.0 0.0 0:00.09 watchdog/1
13 root rt 0 0 0 0 S 0.0 0.0 0:00.13 migration/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.16 ksoftirqd/1
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
19 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
20 root 20 0 0 0 0 S 0.0 0.0 0:00.01 khungtaskd
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
而我们主要关心CPU那一列,输入大写的P可按CPU使用率进行排序,输入q退出,可以看到占用CPU过高的是进程号(PID)为 7933的程序,这个是一个java进程,如无法确定是什么程序,可使用 “ps -fp <进程号>” 进一步确认
第二步,通过ps命令查看CPU占用最高的线程
ps命令用于查看当前进程的快照,命令格式
ps -mp <进程号> -o THREAD,tid,time
-m 表示显示线程
-p 即pid,表示需要显示的进程号
-o 表示显示的列格式,THEAD即线程相关的列,tid即可调度实体的唯一编号,在这里代表线程号,time 即CPU占用时间
输入 ps -mp 7933 -o THREAD,tid,time,得到以下结果,可以得知CPU占用过高的是线程号TID为7955,转为16进制得到1f13 ,在Linux中可使用命令 printf “%x\n” <数字> 进行16进制转换。
[root@localhost jar]# ps -mp 7933 -o THREAD,tid,time
USER %CPU PRI SCNT WCHAN USER SYSTEM TID TIME
root 2.4 - - - - - - 00:04:11
root 0.0 19 - futex_ - - 7933 00:00:00
root 0.0 19 - futex_ - - 7934 00:00:04
root 0.0 19 - futex_ - - 7935 00:00:00
root 0.0 19 - futex_ - - 7936 00:00:00
root 0.0 19 - futex_ - - 7937 00:00:00
root 0.0 19 - futex_ - - 7938 00:00:00
root 0.0 19 - futex_ - - 7939 00:00:00
root 0.0 19 - futex_ - - 7940 00:00:00
root 0.0 19 - futex_ - - 7941 00:00:03
root 0.0 19 - futex_ - - 7942 00:00:01
root 0.0 19 - futex_ - - 7943 00:00:00
root 0.1 19 - futex_ - - 7944 00:00:15
root 0.0 19 - futex_ - - 7951 00:00:01
root 0.0 19 - futex_ - - 7952 00:00:01
root 0.0 19 - futex_ - - 7953 00:00:00
root 0.0 19 - ep_pol - - 7954 00:00:00
root 2.1 19 - - - - 7955 00:03:38
root 0.0 19 - futex_ - - 7956 00:00:00
root 0.0 19 - futex_ - - 7957 00:00:00
root 0.0 19 - futex_ - - 7958 00:00:00
root 0.0 19 - futex_ - - 7959 00:00:00
root 0.0 19 - futex_ - - 7960 00:00:00
root 0.0 19 - futex_ - - 7961 00:00:00
root 0.0 19 - futex_ - - 7962 00:00:00
root 0.0 19 - futex_ - - 7963 00:00:00
root 0.0 19 - futex_ - - 7964 00:00:00
root 0.0 19 - ep_pol - - 7965 00:00:00
root 0.0 19 - inet_c - - 7966 00:00:00
root 0.0 19 - skb_wa - - 9079 00:00:00
第三步,使用jstack定位到具体代码位置
jstack是java查看线程栈的工具,需要安装JDK才有的该命令,命令格式
jstack <进程号> | grep <16进制线程号> -A<显示行数>
-A 即打印出紧随匹配的行之后的下文 N 行
输入 jstack 7933 | grep 1f13 -A50 得到以下结果,可以看到有问题的代码在TestController的第13行,打开代码可知为i++那一行,分析得知原因
[root@localhost jar]# jstack 7933 | grep 1f13 -A50
"http-nio-8080-exec-1" #15 daemon prio=5 os_prio=0 tid=0x00007f37ad179000 nid=0x1f13 runnable [0x00007f379484a000]
java.lang.Thread.State: RUNNABLE
at com.wjq.demo.controller.TestController.test(TestController.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000000f8527df8> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)