【异常】记一次，排查docker容器的微服务，CPU飙高问题

项目时 以docker为容器的 微服务项目。 其中一个docker容器，CPU很高。所以需要排查一下问题。

1. [root@service-slave2 ~]# docker stats

2. CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS NAMES

3. fa4b3663ece4 keda6-slave2-information-plan 733.45% 919MiB / 1.5GiB 59.83% 909MB / 748MB 0B / 0B 217

CPU达到 733%
1、进入容器，查看 进程号

1. [root@service-slave2 ~]# docker exec -it fa4b3663ece4 bash

2. [root@slave2_information_plan project]# top

3. top - 11:13:50 up 11 days, 23:26, 0 users, load average: 44.96, 41.39, 39.79

4. Tasks: 13 total, 1 running, 12 sleeping, 0 stopped, 0 zombie

5. %Cpu(s): 96.8 us, 0.8 sy, 0.0 ni, 2.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

6. KiB Mem : 64757920 total, 571656 free, 28601872 used, 35584392 buff/cache

7. KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29747760 avail Mem

8.  

9. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

10. 8156 root 20 0 6311560 879272 13592 S 706.2 1.4 6122:00 java

11. 1 root 20 0 43272 3072 2392 S 0.0 0.0 0:00.40 systemd

12. 17 root 20 0 47276 9424 9308 S 0.0 0.0 0:00.81 systemd-journal

13. 28 root 20 0 41628 1392 1276 S 0.0 0.0 0:00.15 systemd-udevd

14. 66 root 20 0 26384 1600 1460 S 0.0 0.0 0:00.06 systemd-logind

15. 67 dbus 20 0 58112 2008 1824 S 0.0 0.0 0:00.01 dbus-daemon

16. 81 root 20 0 112924 3524 3332 S 0.0 0.0 0:00.13 sshd

17. 83 root 20 0 9924 724 720 S 0.0 0.0 0:00.00 agetty

18. 8158 root 20 0 13000 828 632 S 0.0 0.0 0:00.48 sh

19. 8172 root 20 0 3131584 72224 12544 S 0.0 0.1 1:05.19 jstatd

20. 15282 root 20 0 15244 2072 1656 S 0.0 0.0 0:00.01 bash

21. 15324 root 20 0 7772 356 280 S 0.0 0.0 0:00.00 sleep

22. 15327 root 20 0 59616 2060 1488 R 0.0 0.0 0:00.00 top

2、使用top -H -p 进程号查看异常线程、查看线程

1. [root@slave2_information_plan project]# top -H -p 8156

2. top - 11:14:20 up 11 days, 23:26, 0 users, load average: 39.80, 40.49, 39.54

3. Threads: 193 total, 30 running, 163 sleeping, 0 stopped, 0 zombie

4. %Cpu(s): 96.0 us, 0.5 sy, 0.0 ni, 3.4 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st

5. KiB Mem : 64757920 total, 540364 free, 28621188 used, 35596368 buff/cache

6. KiB Swap: 33554428 total, 29073788 free, 4480640 used. 29728468 avail Mem

7.  

8. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

9. 14805 root 20 0 6311560 879272 13592 R 43.9 1.4 34:15.23 java

10. 14847 root 20 0 6311560 879272 13592 R 37.5 1.4 29:11.78 java

11. 14848 root 20 0 6311560 879272 13592 R 37.2 1.4 28:24.52 java

12. 14857 root 20 0 6311560 879272 13592 R 35.9 1.4 27:59.42 java

13. 14865 root 20 0 6311560 879272 13592 R 35.5 1.4 27:07.37 java

14. 14692 root 20 0 6311560 879272 13592 R 33.2 1.4 30:21.00 java

15. 14866 root 20 0 6311560 879272 13592 R 29.6 1.4 27:00.60 java

16. 14834 root 20 0 6311560 879272 13592 R 28.6 1.4 28:56.27 java

17. 14833 root 20 0 6311560 879272 13592 R 27.2 1.4 29:11.41 java

18. 14883 root 20 0 6311560 879272 13592 R 26.6 1.4 26:18.58 java

19. 14849 root 20 0 6311560 879272 13592 R 26.2 1.4 28:11.36 java

20. 10125 root 20 0 6311560 879272 13592 R 25.6 1.4 56:53.44 java

21. 14671 root 20 0 6311560 879272 13592 R 24.9 1.4 53:55.84 java

22. 14818 root 20 0 6311560 879272 13592 R 23.9 1.4 31:27.83 java

23. 14873 root 20 0 6311560 879272 13592 R 23.6 1.4 26:25.93 java

24. 8592 root 20 0 6311560 879272 13592 R 23.3 1.4 59:16.66 java

25. 8304 root 20 0 6311560 879272 13592 R 22.3 1.4 63:12.35 java

26. 14685 root 20 0 6311560 879272 13592 R 22.3 1.4 30:09.24 java

27. 8307 root 20 0 6311560 879272 13592 R 21.9 1.4 65:13.92 java

28. 14819 root 20 0 6311560 879272 13592 R 21.9 1.4 30:13.75 java

29. 8310 root 20 0 6311560 879272 13592 R 21.3 1.4 868:16.29 java

30. 8311 root 20 0 6311560 879272 13592 R 19.3 1.4 868:50.86 java

31. 8309 root 20 0 6311560 879272 13592 R 18.9 1.4 872:06.83 java

32. 14864 root 20 0 6311560 879272 13592 R 18.9 1.4 27:53.39 java

33. 8305 root 20 0 6311560 879272 13592 R 18.6 1.4 864:41.14 java

34. 14658 root 20 0 6311560 879272 13592 R 18.3 1.4 59:20.44 java

35. 14693 root 20 0 6311560 879272 13592 R 18.3 1.4 32:41.43 java

36. 8306 root 20 0 6311560 879272 13592 R 14.3 1.4 868:35.26 java

37. 8308 root 20 0 6311560 879272 13592 R 13.0 1.4 861:23.26 java

38. 8175 root 20 0 6311560 879272 13592 S 3.0 1.4 46:57.17 java

39. 15305 root 20 0 6311560 879272 13592 S 0.7 1.4 0:00.55 java

40. 15318 root 20 0 6311560 879272 13592 S 0.3 1.4 0:00.53 java

3、使用printf "%x\n" 线程号将异常线程号转化为16进制

1. [root@slave2_information_plan project]# printf "%x\n" 14805

2. 39d5

4、使用jstack 进程号|grep 16进制异常线程号 -A90来定位异常代码的位置（最后的-A90是日志行数，也可以输出为文本文件或使用其他数字）。可以看到异常代码的位置。

1. [root@slave2_information_plan project]# jstack 8156|grep 39d5 -A90

2. "http-nio-9820-exec-18" #261 daemon prio=5 os_prio=0 tid=0x00007f3e0800f800 nid=0x39d5 runnable [0x00007f3d70580000]

3. java.lang.Thread.State: RUNNABLE

4. at sun.util.calendar.ZoneInfo.getTransitionIndex(ZoneInfo.java:339)

5. at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:259)

6. at sun.util.calendar.ZoneInfo.getOffsets(ZoneInfo.java:236)

7. at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2340)

8. at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)

9. at java.util.Calendar.setTimeInMillis(Calendar.java:1804)

10. at java.util.GregorianCalendar.add(GregorianCalendar.java:1076)

11. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.longOfTwoDate(CardPlanServiceJHImpl.java:1969)

12. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl.restartCardPlanZZZ(CardPlanServiceJHImpl.java:2272)

13. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$FastClassBySpringCGLIB$$26beb7de.invoke(<generated>)

14. at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

15. at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)

16. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

17. at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)

18. at com.keda.util.log.LogRecordAspect.doAroundForService(LogRecordAspect.java:233)

19. at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)

20. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

21. at java.lang.reflect.Method.invoke(Method.java:498)

22. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)

23. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)

24. at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

25. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

26. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)

27. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

28. at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

29. at com.keda.information.plan.service.impl.CardPlanServiceJHImpl$$EnhancerBySpringCGLIB$$115419c6.restartCardPlanZZZ(<generated>)

30. at com.keda.information.plan.service.api.ICardPlanServiceApi.restartCardPlanZ(ICardPlanServiceApi.java:259)

31. at com.keda.information.plan.service.api.ICardPlanServiceApi$$FastClassBySpringCGLIB$$6b2eaa41.invoke(<generated>)

32. at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

33. at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)

34. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

35. at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)

36. at com.keda.util.log.LogRecordAspect.doAroundForApi(LogRecordAspect.java:177)

37. at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)

38. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

39. at java.lang.reflect.Method.invoke(Method.java:498)

40. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)

41. at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)

42. at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

43. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

44. at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)

45. at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)

46. at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

47. at com.keda.information.plan.service.api.ICardPlanServiceApi$$EnhancerBySpringCGLIB$$c22f468b.restartCardPlanZ(<generated>)

48. at sun.reflect.GeneratedMethodAccessor867.invoke(Unknown Source)

49. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

50. at java.lang.reflect.Method.invoke(Method.java:498)

51. at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)

52. at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)

53. at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)

54. at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)

55. at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)

56. at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

57. at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)

58. at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)

59. at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)

60. at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)

61. at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)

62. at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)

63. at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

64. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

65. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

66. at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

67. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

68. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

69. at brave.servlet.TracingFilter.doFilter(TracingFilter.java:65)

70. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

71. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

72. at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:124)

73. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

74. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

75. at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88)

76. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

77. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

78. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

79. at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

80. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

81. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

82. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

83. at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)

84. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

85. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

86. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

87. at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)

88. at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

89. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

90. at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

91. at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:50)

92. at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

注意事项：

如果提示 -bash: jstack: command not found，则可以找到jdk的目录，进入bin目录，

        在执行 ./jstack 进程号 | grep 16进制异常线程号 -A90 即可。
如果提示 well-known file is not secure，则是因为账号权限的问题。在/tmp/hsperfdata_$USER/目录，有一个以进程号命名的文件，当我们执行jmap或者jstack出现上叙信息时，先检查执行该命令的用户是否和hsperfdata_$USER这个文件所属的用户一致，如果不一致，切换至成一致再执行。执行：sudo -u 有权限的账号 ./jstack 进程号 | grep 16进制异常线程号 -A50 即可。

5、查看代码

1. public static int longOfTwoDate(Date first, Date second) throws ParseException {

2. Calendar calendar = Calendar.getInstance();

3. calendar.setTime(first);

4. int cnt = 0;

5. while (calendar.getTime().compareTo(second) != 0) {

6. calendar.add(Calendar.DATE, 1);

7. cnt++;

8. }

9. return cnt;

10.  

11. }

发现一个 while的死循环。修改如下：

1. public static int longOfTwoDate(Date first, Date second) throws ParseException {

2. Calendar calendar = Calendar.getInstance();

3. calendar.setTime(first);

4. int cnt = 0;

5. if(calendar.getTime().compareTo(second) <= 0){

6. while (calendar.getTime().compareTo(second) != 0) {

7. calendar.add(Calendar.DATE, 1);

8. cnt++;

9. }

10. }

11. return cnt;

12.  

13. }