1、系统出现无法正常访问现象、重启tomcat后系统恢复、但等一会儿系统又无法正常访问了
2、linux服务器排查
使用top命令查看资源占用情况,发现pid为xxx的进程占用了大量的CPU资源,CPU占用率高达100%,总共CPU使用300%、
示例
[ylp@ylp-web-01 ~]$ top
top - 14:51:10 up 233 days, 11:40, 7 users, load average: 6.85, 5.62, 3.97
Tasks: 192 total, 2 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 97.3 us, 0.3 sy, 0.0 ni, 2.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16268652 total, 5114392 free, 6907028 used, 4247232 buff/cache
KiB Swap: 4063228 total, 3989708 free, 73520 used. 8751512 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14063 ylp 20 0 9260488 4.627g 11976 S 776.1 29.8 117:41.66 java
2、定位问题线程查看线程占用cpu情况、找出占用cpu较高的线程id
top -p xxx-H
3、查看问题线程堆栈
挑选TID为24607的线程,查看该线程的堆栈情况,先将线程id转为16进制,使用printf "%x\n" tid命令进行转换
printf "%x\n" 24607
再使用jstack命令打印线程堆栈信息,命令格式:jstack pid |grep tid -A 30
[ylp@ylp-web-01 ~]$ jstack 14063 |grep 36f1 -A 30
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fa35001e800 nid=0x36f1 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fa350020800 nid=0x36f2 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fa350022800 nid=0x36f3 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fa350024000 nid=0x36f4 runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007fa350026000 nid=0x36f5 runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007fa350028000 nid=0x36f6 runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007fa350029800 nid=0x36f7 runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007fa35002b800 nid=0x36f8 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fa3500a8800 nid=0x3700 waiting on condition
JNI global references: 392
4、发现有大量虚拟机GC现在应查询java进程内存使用情况
使用命令 jmap -heap xxx
发现新生代内存使用过高 修改tomcat内存配置将MaxNewSize 增加重启tomcat后 查看进程内存使用发现新生代内存使用率正常问题解决。