记录一次生产应用启动后发生多次FullGc的解决过程

昨天生产发布完后，收到告警短信5分钟内FullGc次数大于2次。当时一脸懵，应用上线前做过压力测试，没测出jvm fullgc问题啊。所以按照告警时间去查日志，发现基本为spring初始化动作。

后面过了半个小时也没有收到告警，基本猜测是meta区的问题

下面是定位过程。

1,首先ps -ef|grep java
cd /opt/wildfly/openjdk/openjdk-1.8.0_92/bin

2,[root@uqotsxgs1a01 bin]# ./jmap -heap 111255
Attaching to process ID 111255, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.92-b00

using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 4294967296 (4096.0MB)
NewSize = 174456832 (166.375MB)
MaxNewSize = 174456832 (166.375MB)
OldSize = 4120510464 (3929.625MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)（初始meta大小为20M）
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)

3,查看实际meta大小
[root@uqotsxgs1a01 bin]# ./jstat -gccapacity 111255
NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC 
170368.0 170368.0 170368.0 17024.0 17024.0 136320.0 4023936.0 4023936.0 4023936.0 4023936.0 0.0 1202176.0 172332.0 0.0 1048576.0 19756.0 1951 7
[root@uqotsxgs1a01 bin]# ./jstat -gc 111255
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 
17024.0 17024.0 0.0 938.0 136320.0 13046.0 4023936.0 225361.8 172332.0 160840.8 19756.0 16833.0 1951 61.236 7 0.799 62.035

而 实际 meta可用为173M，实际已用160M，Fullgc次数为 7.。所以问题很明显，随着加载类的增多，我们的应用meta区远超过20m的大小，所以会触发fullgc，扩容metaspace。至此，问题已经很明确，因为jdk1.8的默认参数，导致需要扩容meta区，因此出现了fullgc。

4，修改meta区大小，重启服务
JAVA_OPTS="$JAVA_OPTS -Xms4096m -Xmx4096m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m"

5，查看修改的Fullgc次数 为 1，解决。
[root@uqotsxgs1a01 bin]# ./jstat -gc 35656
S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 
17024.0 17024.0 3228.8 0.0 136320.0 100896.4 4023936.0 195764.5 159616.0 150182.8 18816.0 16251.0 84 4.365 1 0.578 4.943