JVM中的ExplicitGCInvokesConcurrent选项

问题描述：

最近经常收到Hbase的FGC报警，但看gc日志如下：

2019-12-24T14:33:12.332+0800: 9446402.403: [Full GC (System.gc()) 2019-12-24T14:33:12.332+0800: 9446402.404: [CMS: 9746276K->9749022K(44040192K), 1.8478848 secs] 9762182K->9749022K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.8536369 secs] [Times: user=1.86 sys=0.00, real=1.86 secs]
2019-12-24T14:33:27.973+0800: 9446418.045: [Full GC (System.gc()) 2019-12-24T14:33:27.974+0800: 9446418.045: [CMS: 9749022K->9741506K(44040192K), 2.9043246 secs] 10310705K->9741506K(45927680K), [Metaspace: 55550K->55550K(57344K)], 2.9099166 secs] [Times: user=2.92 sys=0.00, real=2.91 secs]
2019-12-24T14:33:42.051+0800: 9446432.122: [Full GC (System.gc()) 2019-12-24T14:33:42.051+0800: 9446432.122: [CMS: 9741506K->9734271K(44040192K), 1.7758858 secs] 10289122K->9734271K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.7813927 secs] [Times: user=1.78 sys=0.00, real=1.78 secs]
2019-12-24T14:34:01.748+0800: 9446451.820: [Full GC (System.gc()) 2019-12-24T14:34:01.749+0800: 9446451.820: [CMS: 9734271K->9887663K(44040192K), 3.0752197 secs] 10641723K->9887663K(45927680K), [Metaspace: 55550K->55550K(57344K)], 3.0813152 secs] [Times: user=3.08 sys=0.00, real=3.08 secs]
2019-12-24T14:34:05.479+0800: 9446455.550: [Full GC (System.gc()) 2019-12-24T14:34:05.479+0800: 9446455.550: [CMS: 9887663K->9868928K(44040192K), 1.6589594 secs] 10027519K->9868928K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.6645320 secs] [Times: user=1.67 sys=0.00, real=1.66 secs]
2019-12-24T14:34:07.893+0800: 9446457.964: [Full GC (System.gc()) 2019-12-24T14:34:07.893+0800: 9446457.965: [CMS: 9868928K->9869164K(44040192K), 1.8132781 secs] 10012277K->9869164K(45927680K), [Metaspace: 55550K->55550K(57344K)], 1.8188013 secs] [Times: user=1.82 sys=0.00, real=1.82 secs]8 secs]

hbase日志如下：

2019-12-24 14:34:12,950 WARN  [hconnection-0x239ab184-shared--pool4-t2855938] client.AsyncProcess: #79, table=auditor_default, attempt=4/4 failed=96ops, last exception: java.io.IOException: com.google.protobuf.ServiceException: java.lang.OutOfMemoryError: Direct buffer memory on hbase007,16020,1565365569654, tracking started Tue Dec 24 14:34:01 CST 2019; not retrying 96 - final failure

问题分析：

由于由于系统大量使用了NIO中的DirectByteBuffer（hbase regionserver设置了堆外内存DirectoryBuffer为20G)，需要定期清理本地内存。
DirectByteBuffer通过内存映射，使java进程直接访问与文件相关联的虚拟地址空间，减少了文件拷贝带来的开销，提高了文件读取效率。这一块虚拟地址空间并不是分配在jvm堆上，而是分配在native堆上。yong gc不能回收这部分空间，只能通过Full gc顺带进行回收。

从上面的日志可以看出每次Full gc，所花时间将近2秒，这2秒是停机回收，系统会有卡顿存在，用户体验不太好。

解决方法：

希望Full gc有更少的停机时间，可以启用-XX:+ExplicitGCInvokesConcurrent或-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses选项。