Uber jvm-profiler学习

Build

git clone git@github.com:uber-common/jvm-profiler.git
https://github.com/wankunde/jvm-profiler.git (自己实现的Graphite Metrics Reporter)
mvn clean package -DskipTests=true -T2C

系统实现原理

1. jvm 启用Agent
2. Agent 启动配置的reporter 和 profilers
3. 在每个profiler内部通过 ManagementFactory 构造MXBean 获取程序内部状态信息
4. 通过Report 发送metrics

Java Agent Overview

详细实现

• 通过maven 的maven-jar-plugin 插件在结果jar包中指定 agent class

		<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.6</version>
                <configuration>
                    <archive>
                        <manifestEntries>
                            <Agent-Class>com.uber.profiling.Agent</Agent-Class>
                            <Premain-Class>com.uber.profiling.Agent</Premain-Class>
                        </manifestEntries>
                    </archive>
                </configuration>
            </plugin>

• Agent 代码实现了agentmain 和 premain 方法

public final class Agent {

    private static AgentImpl agentImpl = new AgentImpl();

    private Agent() {
    }

    public static void agentmain(final String args, final Instrumentation instrumentation) {
        premain(args, instrumentation);
    }

    public static void premain(final String args, final Instrumentation instrumentation) {
        System.out.println("Java Agent " + AgentImpl.VERSION + " premain args: " + args);

        Arguments arguments = Arguments.parseArgs(args);
        arguments.runConfigProvider();
        agentImpl.run(arguments, instrumentation, null);
    }
}

• 在程序运行时加入agent jar包

    -javaagent:/Users/wankun/ws/wankun/jvm-profiler/target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,tag=tag1,metricInterval=10000,durationProfiling=com.uber.profiling.examples.HelloWorldApplication.*
  

Profile Application

因为公司环境原因，本文使用了我自己实现的Graphite Reporter(com.uber.profiling.reporters.GraphiteOutputReporter) 将数据写入到Graphite中再用 Grafana 来展示，其他Reporter 类似。

Profile Java Application

• 设置系统环境

    export test_app_id=hello_jvm_profile
  

• 程序启动的时候指定使用哪一个系统变量作为appId

程序启动的时候可以把profile参数写在命令行，也可以将参数写在配置文件中

• 命令行方式

    java -javaagent:target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.GraphiteOutputReporter,configProvider=com.uber.profiling.YamlConfigProvider,configFile=../graphite.yaml,tag=mytag2,appIdVariable=test_app_id,metricInterval=5000,durationProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod,argumentProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod.1,sampleInterval=5000 -cp target/jvm-profiler-1.0.0.jar com.uber.profiling.examples.HelloWorldApplication
  

• 配置文件方式

    java -javaagent:target/jvm-profiler-1.0.0.jar=configProvider=com.uber.profiling.YamlConfigProvider,configFile=../graphite.yaml -cp target/jvm-profiler-1.0.0.jar com.uber.profiling.examples.HelloWorldApplication
  

graphite.yaml

reporter:
  com.uber.profiling.reporters.GraphiteOutputReporter

tag:
  mytag2

appIdVariable:
  test_app_id

metricInterval:
  5000

sampleInterval:
  5000

graphite:
  host: 127.0.0.1
  port: 2003
  prefix: jvm

durationProfiling:
  com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod

argumentProfiling:
  com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod.1

• 目前metrics 生成格式为：

    jvm.spark2.${appId}.${HOST}.*
  

appId : 我们设置的应用名称
HOST : JVM 启动的机器名

Profile Spark Application

如果是对spark 程序采用

• 上传编译好的Jar 到 HDFS

    hdfs dfs -put target/jvm-profiler-1.0.0.jar /user/kun.wan/
  

• 启动测试Spark程序

  • 命令行填充profile参数

spark2-submit --deploy-mode cluster --master yarn \
    --files graphite.yaml \
    --conf spark.jars=hdfs:///user/kun.wan/jvm-profiler-1.0.0.jar \
    --conf spark.driver.extraJavaOptions='-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.GraphiteOutputReporter,configProvider=com.uber.profiling.YamlConfigProvider,configFile=graphite.yaml,tag=spark2,metricInterval=5000,sampleInterval=5000' \
    --conf spark.executor.extraJavaOptions='-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.GraphiteOutputReporter,configProvider=com.uber.profiling.YamlConfigProvider,configFile=graphite.yaml,tag=spark2,metricInterval=5000,sampleInterval=5000' \
    --class org.apache.spark.examples.SparkPi \
    --num-executors 3 \
    --driver-memory 1024m \
    --executor-memory 1024m \
    --executor-cores 1 \
    /opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/examples/jars/spark-examples_2.11-2.4.0.cloudera2.jar 100000

• 通过配置文件填充profile参数

spark2-submit --deploy-mode cluster --master yarn \
        --files spark2_graphite.yaml \
        --conf spark.jars=hdfs:///user/kun.wan/jvm-profiler-1.0.0.jar \
        --conf spark.driver.extraJavaOptions='-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.GraphiteOutputReporter,configProvider=com.uber.profiling.YamlConfigProvider,configFile=spark2_graphite.yaml' \
        --conf spark.executor.extraJavaOptions='-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.GraphiteOutputReporter,configProvider=com.uber.profiling.YamlConfigProvider,configFile=spark2_graphite.yaml' \
        --class org.apache.spark.examples.SparkPi \
        --num-executors 3 \
        --driver-memory 1024m \
        --executor-memory 1024m \
        --executor-cores 1 \
        /opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/examples/jars/spark-examples_2.11-2.4.0.cloudera2.jar 100000

reporter:
  com.uber.profiling.reporters.GraphiteOutputReporter

tag:
  spark2

metricInterval:
  5000

sampleInterval:
  5000

graphite:
  host: 127.0.0.1
  port: 2003
  prefix: java

• 程序生成的Metrics格式和上面一样，不过appId 会有程序自动获取Yarn的application_id 进行填充（PS: 如果在一台机器上有两个Executor在运行，生成的Metrics可能有问题）

Profile Java Application 并解读Metrics数据

java -javaagent:target/jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.ConsoleOutputReporter,tag=mytag,metricInterval=5000,durationProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod,argumentProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod.1,sampleInterval=100 -cp target/jvm-profiler-1.0.0.jar com.uber.profiling.examples.HelloWorldApplication

-javaagent:target/jvm-profiler-1.0.0.jar : 启用jvm-profile
reporter=com.uber.profiling.reporters.ConsoleOutputReporter : 测试使用console output
tag=mytag : output tag
metricInterval=5000 : 采样时间间隔

durationProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod
argumentProfiling=com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod.1
sampleInterval=100

MethodDuration

durationProfiling 会对指定的方法执行时间进行统计，包括方法的执行次数，执行总耗时，最长耗时，最短耗时

ConsoleOutputReporter - MethodDuration: {
    "metricName": "duration.count",
    "processName": "67146@wankunMBP.local",
    "appId": null,
    "host": "wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "metricValue": 2,
    "methodName": "publicSleepMethod",
    "className": "com.uber.profiling.examples.HelloWorldApplication",
    "epochMillis": 1568772161319,
    "tag": "mytag"
}
ConsoleOutputReporter - MethodDuration: {
    "metricName": "duration.sum",
    "processName": "67146@wankunMBP.local",
    "appId": null,
    "host": "wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "metricValue": 2237,
    "methodName": "publicSleepMethod",
    "className": "com.uber.profiling.examples.HelloWorldApplication",
    "epochMillis": 1568772161319,
    "tag": "mytag"
}
ConsoleOutputReporter - MethodDuration: {
    "metricName": "duration.min",
    "processName": "67146@wankunMBP.local",
    "appId": null,
    "host": "wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "metricValue": 1077,
    "methodName": "publicSleepMethod",
    "className": "com.uber.profiling.examples.HelloWorldApplication",
    "epochMillis": 1568772161319,
    "tag": "mytag"
}
ConsoleOutputReporter - MethodDuration: {
    "metricName": "duration.max",
    "processName": "67146@wankunMBP.local",
    "appId": null,
    "host": "wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "metricValue": 1160,
    "methodName": "publicSleepMethod",
    "className": "com.uber.profiling.examples.HelloWorldApplication",
    "epochMillis": 1568772161319,
    "tag": "mytag"
}

MethodArgument

argumentProfiling 对执行的方法的参数进行采样

ConsoleOutputReporter - MethodArgument: {
    "metricName": "arg.1.1158",
    "processName": "67146@wankunMBP.local",
    "appId": null,
    "host": "wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "metricValue": 1,
    "methodName": "publicSleepMethod",
    "className": "com.uber.profiling.examples.HelloWorldApplication",
    "epochMillis": 1568772161321,
    "tag": "mytag"
}
ConsoleOutputReporter - MethodArgument: {
    "metricName": "arg.1.1073",
    "processName": "67146@wankunMBP.local",
    "appId": null,
    "host": "wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "metricValue": 1,
    "methodName": "publicSleepMethod",
    "className": "com.uber.profiling.examples.HelloWorldApplication",
    "epochMillis": 1568772161321,
    "tag": "mytag"
}

Stacktrace

Stacktrace不知道是不是默认输出的？
这里给了5个StackTrace 的sample，有2个时间点采用到了main 线程，2个采样到了jvm thread，还有1个采样到空

ConsoleOutputReporter - Stacktrace: {
    "stacktrace": [
        "java.lang.Thread.sleep",
        "com.uber.profiling.examples.HelloWorldApplication.privateSleepMethod",
        "com.uber.profiling.examples.HelloWorldApplication.main"
    ],
    "endEpoch": 1568772161323,
    "appId": null,
    "host": "wankunMBP.local",
    "name": "67146@wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "threadState": "TIMED_WAITING",
    "count": 28,
    "tag": "mytag",
    "startEpoch": 1568772156323,
    "threadName": "main"
}
ConsoleOutputReporter - Stacktrace: {
    "stacktrace": [
        "java.lang.Thread.sleep",
        "com.uber.profiling.examples.HelloWorldApplication.publicSleepMethod",
        "com.uber.profiling.examples.HelloWorldApplication.main"
    ],
    "endEpoch": 1568772161323,
    "appId": null,
    "host": "wankunMBP.local",
    "name": "67146@wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "threadState": "TIMED_WAITING",
    "count": 22,
    "tag": "mytag",
    "startEpoch": 1568772156323,
    "threadName": "main"
}
ConsoleOutputReporter - Stacktrace: {
    "stacktrace": [
        "java.lang.Object.wait",
        "java.lang.ref.ReferenceQueue.remove",
        "java.lang.ref.ReferenceQueue.remove",
        "java.lang.ref.Finalizer$FinalizerThread.run"
    ],
    "endEpoch": 1568772161323,
    "appId": null,
    "host": "wankunMBP.local",
    "name": "67146@wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "threadState": "WAITING",
    "count": 50,
    "tag": "mytag",
    "startEpoch": 1568772156323,
    "threadName": "Finalizer"
}
ConsoleOutputReporter - Stacktrace: {
    "stacktrace": [],
    "endEpoch": 1568772161323,
    "appId": null,
    "host": "wankunMBP.local",
    "name": "67146@wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "threadState": "RUNNABLE",
    "count": 50,
    "tag": "mytag",
    "startEpoch": 1568772156323,
    "threadName": "Signal Dispatcher"
}
ConsoleOutputReporter - Stacktrace: {
    "stacktrace": [
        "java.lang.Object.wait",
        "java.lang.Object.wait",
        "java.lang.ref.Reference.tryHandlePending",
        "java.lang.ref.Reference$ReferenceHandler.run"
    ],
    "endEpoch": 1568772161323,
    "appId": null,
    "host": "wankunMBP.local",
    "name": "67146@wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "threadState": "WAITING",
    "count": 50,
    "tag": "mytag",
    "startEpoch": 1568772156323,
    "threadName": "Reference Handler"
}

CpuAndMemory

先看一下Memory

Heap & NonHeap

JVM通过MemoryUsage实际上可以得到三个内存使用的指标 :Used, Commmitted, Max，还是看MemoryUsage Java Doc 靠谱，下面引用部分

getUsed is:

the amount of used memory in bytes

getCommitted()

Returns the amount of memory in bytes that is committed for the Java virtual machine to use. This amount of memory is guaranteed for the Java virtual machine to use.

getMax()

Returns the maximum amount of memory in bytes that can be used for memory management. This method returns -1 if the maximum memory size is undefined.

This amount of memory is not guaranteed to be available for memory management if it is greater than the amount of committed memory. The Java virtual machine may fail to allocate memory even if the amount of used memory does not exceed this maximum size.

示意图

    |--------|
       init
    |---------------|
           used
    |---------------------------|
              committed
    |----------------------------------------------|
                        max

JVM max使用内存，一般都是我们程序启动时指定，且不会变化，所以不需要额外关注。同时jvm内存又分为 heap 和 non-heap，所以我们重点关注下面四个指标：

"nonHeapMemoryCommitted": 16449536,
"nonHeapMemoryTotalUsed": 15809792,
"heapMemoryTotalUsed": 35252640,
"heapMemoryCommitted": 257425408,

JVM Memory & Process Memory

• 进程实际占用物理内存和虚拟内存

PS查看

ps aux |grep $PID
USER PID %CPU %MEM VSZ RSS
hdfs 7664 10.2 2.4 6238460 1615368
hdfs 7674 0.0 0.0 354560 7784 ? Sl Oct31 0:00 python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-redactor /usr/lib64/cmf/service/hdfs/hdfs.sh namenode

说明 :
VSZ ： 虚拟内存使用，6238460 KB = 6388183040 B
RSS： 物理内存使用，1615368 KB = 1654136832 B 约 1.6G

Top 查看

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7664 hdfs 20 0 6238460 1.5g 13572 S 13.3 2.5 2659:12 java

这个结果和上面一致

stat 文件查看

-bash-4.2# cat /proc/7664/stat
7664 (java) S 23537 7664 23537 0 -1 1077944576 1270462 5470849 7 1 9114605 6839303 1031 1806 20 0 144 0 677316526 6388183040 403842 18446744073709551615 4194304 4196284 140733787380384 140733787363056 140017828220743 0 0 0 16800975 18446744073709551615 0 0 17 13 0 0 0 0 0 6294936 6295608 23248896 140733787385041 140733787386032 140733787386032 140733787394002 0

解析结果

name:(java)
ppid:23537
pgrpId:7664
session:23537
utime:9067085
stime:6805494
vsize:6388183040
rss:403837 * 4096(页大小) = 1654116352

-bash-4.2# getconf PAGESIZE
4096

说明： ps aux 会从 /proc目录下读取对应的 stat 文件，stat 文件内容解析如下（hadoop通过监控此文件来实现内存和CPU的监控)

其中vsize 和上面的进程使用的虚拟内存完全一致，rss 的值表示 page个数，看到 PAGESIZE = 4096 ,所以计算结果和上面基本对的上

• Heap 内存使用

因为Heap的内存设计到heap 对象分配和GC，取的值一直处于变化中。但是可以很明显的观察到我们使用的Heap内存的峰值并未到达Process RSS 内存的使用。

Memory                                                              used                  total                  max                   usage                  GC
heap                                                                777M                  3987M                  3987M                 19.49%                 gc.parnew.count                                                                11667
par_eden_space                                                      515M                  865M                   865M                  59.61%                 gc.parnew.time(ms)                                                             146092
par_survivor_space                                                  58M                   108M                   108M                  54.34%                 gc.concurrentmarksweep.count                                                   2
cms_old_gen                                                         202M                  3014M                  3014M                 6.72%                  gc.concurrentmarksweep.time(ms)                                                214
nonheap                                                             118M                  120M                   -1                    98.05%
code_cache                                                          47M                   47M                    240M                  19.64%
metaspace                                                           64M                   65M                    -1                    98.05%
compressed_class_space                                              7M                    7M                     1024M                 0.69%
direct                                                              20M                   20M                    -                     100.00%
mapped                                                              0K                    0K                     -                     NaN%

Jmap 结果查看

-bash-4.2# jmap -heap 7664
Attaching to process ID 7664, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.192-b12

using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 4294967296 (4096.0MB)
   NewSize                  = 1134100480 (1081.5625MB)
   MaxNewSize               = 1134100480 (1081.5625MB)
   OldSize                  = 3160866816 (3014.4375MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 1020723200 (973.4375MB)
   used     = 837543704 (798.7439193725586MB)
   free     = 183179496 (174.6935806274414MB)
   82.05394998369783% used
Eden Space:
   capacity = 907345920 (865.3125MB)
   used     = 795390736 (758.5437164306641MB)
   free     = 111955184 (106.76878356933594MB)
   87.66124566912694% used
From Space:
   capacity = 113377280 (108.125MB)
   used     = 42152968 (40.20020294189453MB)
   free     = 71224312 (67.92479705810547MB)
   37.179378443370666% used
To Space:
   capacity = 113377280 (108.125MB)
   used     = 0 (0.0MB)
   free     = 113377280 (108.125MB)
   0.0% used
concurrent mark-sweep generation:
   capacity = 3160866816 (3014.4375MB)
   used     = 211925960 (202.10834503173828MB)
   free     = 2948940856 (2812.3291549682617MB)
   6.704678568779027% used


21720 interned Strings occupying 2098528 bytes.

我的理解是Jvm的物理内存占用有一个担保值（committed），这个值会大于JVM的最大内存占用，committed值才是jvm进程实际向操作系统申请的内存。

Java Buffer Pool

在 java.nio.* 包中引入的Buffer Pool 包含两种Byte Buffer

• Non-Direct Buffer : 通过 ByteBuffer.allocate() and ByteBuffer.wrap() 方法来分配 on-heap buffer
  • 操作系统执行的I/O 指令是一堆连续的bytes序列
  • JVM规范不担保 allocate() 的buffer在内存中是连续的，（JVM规范对自己 Heap space 在内存中也不能包装是连续分配的）。虽然JVM 看起来不太可能会将基础数据类型的一维数组放在内存中的不同位置，但是JVM Heap 中的 byte array仍然不能被本地 I/O 指令直接访问访问，每次 I/O 操作之前需要先将数据copy 到本地内存，并导致非常低效。因此引入了 Direct Buffer
• Direct Buffer :
  • native memory 中和Java共享的一块内存，可以直接访问；
  • 通过ByteBuffer.allocateDirect() 方法生成 DirectByteBuffer 实例；
  • 可以非常高效的执行 I/O 操作
• Memory Mapped Buffer
  • 映射文件的一个区域到内存中(堆外，GC无法回收，由操作系统管理)，后续可以直接访问;
  • 对于需要多次访问文件的操作，非常有用，后续对文件会直接读取内存中的数据，无需再访问磁盘;
  • 通过 FileChannel.map()方法创建MappedByteBuffer实例；
• Allocation is Expensive
  • DirectBuffer 分配非常昂贵
  • 如果每次 slice 的大小不一样，有可能会造成碎片话(frqgmented)
  • buffer 不能被 compacted

可以看到，这里帮我们监控了 direct 和 mapped 两种类型buffer的使用大小

memoryPools

• PS Eden Space : The pool from which memory is initially allocated for most objects.
• PS Survivor Space : The pool containing objects that have survived the garbage collection of the Eden space.
• PS Old Gen : The pool containing objects that have existed for some time in the survivor space.
• Code Cache : The HotSpot Java VM also includes a code cache, containing memory that is used for compilation and storage of native code.
• Metaspace : The OpenJDK uses Metaspace to store its class metadata. It can contribute a large part to the non-Java-heap memory footprint of a Java VM process.
• Compressed Class Space : Compressed Object Pointers (“CompressedOops”) and Compressed Class Pointers.

GC

• GC 类型
• GC 时间
• GC次数

cpu load

• 当前线程 load
• 当前系统 load

操作系统信息

ConsoleOutputReporter - CpuAndMemory: 
{
    "epochMillis": 1568772161326,
    "bufferPools": [
        {
            "totalCapacity": 0,
            "name": "direct",
            "count": 0,
            "memoryUsed": 0
        },
        {
            "totalCapacity": 0,
            "name": "mapped",
            "count": 0,
            "memoryUsed": 0
        }
    ],

    "nonHeapMemoryCommitted": 16449536,
    "nonHeapMemoryTotalUsed": 15809792,
    "heapMemoryTotalUsed": 35252640,
    "heapMemoryCommitted": 257425408,
    "memoryPools": [
        {
            "peakUsageMax": 251658240,
            "usageMax": 251658240,
            "peakUsageUsed": 2429632,
            "name": "Code Cache",
            "peakUsageCommitted": 2555904,
            "usageUsed": 2426304,
            "type": "Non-heap memory",
            "usageCommitted": 2555904
        },
        {
            "peakUsageMax": -1,
            "usageMax": -1,
            "peakUsageUsed": 12048080,
            "name": "Metaspace",
            "peakUsageCommitted": 12451840,
            "usageUsed": 12048080,
            "type": "Non-heap memory",
            "usageCommitted": 12451840
        },
        {
            "peakUsageMax": 1073741824,
            "usageMax": 1073741824,
            "peakUsageUsed": 1335408,
            "name": "Compressed Class Space",
            "peakUsageCommitted": 1441792,
            "usageUsed": 1335408,
            "type": "Non-heap memory",
            "usageCommitted": 1441792
        },
        {
            "peakUsageMax": 1409286144,
            "usageMax": 1409286144,
            "peakUsageUsed": 35252640,
            "name": "PS Eden Space",
            "peakUsageCommitted": 67108864,
            "usageUsed": 35252640,
            "type": "Heap memory",
            "usageCommitted": 67108864
        },
        {
            "peakUsageMax": 11010048,
            "usageMax": 11010048,
            "peakUsageUsed": 0,
            "name": "PS Survivor Space",
            "peakUsageCommitted": 11010048,
            "usageUsed": 0,
            "type": "Heap memory",
            "usageCommitted": 11010048
        },
        {
            "peakUsageMax": 2863661056,
            "usageMax": 2863661056,
            "peakUsageUsed": 0,
            "name": "PS Old Gen",
            "peakUsageCommitted": 179306496,
            "usageUsed": 0,
            "type": "Heap memory",
            "usageCommitted": 179306496
        }
    ],
    "processCpuLoad": 0.0018559132616109244,
    "systemCpuLoad": 0.0521978021978022,
    "processCpuTime": 1297096000,
    "appId": null,
    "name": "67146@wankunMBP.local",
    "host": "wankunMBP.local",
    "processUuid": "a2b891d1-e0de-4424-83ba-23c46d177570",
    "tag": "mytag",
    "gc": [
        {
            "collectionTime": 0,
            "name": "PS Scavenge",
            "collectionCount": 0
        },
        {
            "collectionTime": 0,
            "name": "PS MarkSweep",
            "collectionCount": 0
        }
    ]
}

JVM最大内存

• 通常来说，我们通过设置JAVA程序的-Xmx 参数来设置JVM的最大值，但是JVM在运行的时候实际可以使用内存空间还需要在该值的基础上减去其中一个Survivor空间的。
• 获取JVM最大可用内存
  • 通过Java代码Runtime.getRuntime().maxMemory()获取
  • 使用JMC工具的Java Bean来获取
  • 通过arthas的dashboard查看JVM的ON_HEAP和OFF_HEAP的值大小
  • 通过 jmap -heap [PID] 查看各个区的使用值
  • 生产环境不要用-XX:-UseAdaptiveSizePolicy ，因为JVM可以动态调整各个区的大小，要更加灵活。
• 以2G内存的JVM 查看分析结果，程序启动参数:-Xmx2048m -XX:+PrintHeapAtGC -XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=8
  
• 通过参数-XX:+PrintHeapAtGC在打印GC日志，观察到的JVM Pool Size是不准的，我的理解是这里显示的应该是各个Pool 的Used 和 Commited内存，而不是Max内存。

{Heap before GC invocations=457 (full 228):
 PSYoungGen      total 78336K, used 7198K [0x0000000795580000, 0x000000079aa80000, 0x00000007c0000000)
  eden space 69632K, 10% used [0x0000000795580000,0x0000000795c87b10,0x0000000799980000)
  from space 8704K, 0% used [0x000000079a200000,0x000000079a200000,0x000000079aa80000)
  to   space 8704K, 0% used [0x0000000799980000,0x0000000799980000,0x000000079a200000)
 ParOldGen       total 175104K, used 6240K [0x0000000740000000, 0x000000074ab00000, 0x0000000795580000)
  object space 175104K, 3% used [0x0000000740000000,0x0000000740618020,0x000000074ab00000)
 Metaspace       used 10066K, capacity 10572K, committed 10752K, reserved 1058816K
  class space    used 1151K, capacity 1252K, committed 1280K, reserved 1048576K
Heap after GC invocations=457 (full 228):
 PSYoungGen      total 78336K, used 224K [0x0000000795580000, 0x000000079aa80000, 0x00000007c0000000)
  eden space 69632K, 0% used [0x0000000795580000,0x0000000795580000,0x0000000799980000)
  from space 8704K, 2% used [0x0000000799980000,0x00000007999b8000,0x000000079a200000)
  to   space 8704K, 0% used [0x000000079a200000,0x000000079a200000,0x000000079aa80000)
 ParOldGen       total 175104K, used 6240K [0x0000000740000000, 0x000000074ab00000, 0x0000000795580000)
  object space 175104K, 3% used [0x0000000740000000,0x0000000740618020,0x000000074ab00000)
 Metaspace       used 10066K, capacity 10572K, committed 10752K, reserved 1058816K
  class space    used 1151K, capacity 1252K, committed 1280K, reserved 1048576K
}
{Heap before GC invocations=458 (full 229):
 PSYoungGen      total 78336K, used 224K [0x0000000795580000, 0x000000079aa80000, 0x00000007c0000000)
  eden space 69632K, 0% used [0x0000000795580000,0x0000000795580000,0x0000000799980000)
  from space 8704K, 2% used [0x0000000799980000,0x00000007999b8000,0x000000079a200000)
  to   space 8704K, 0% used [0x000000079a200000,0x000000079a200000,0x000000079aa80000)
 ParOldGen       total 175104K, used 6240K [0x0000000740000000, 0x000000074ab00000, 0x0000000795580000)
  object space 175104K, 3% used [0x0000000740000000,0x0000000740618020,0x000000074ab00000)
 Metaspace       used 10066K, capacity 10572K, committed 10752K, reserved 1058816K
  class space    used 1151K, capacity 1252K, committed 1280K, reserved 1048576K
Heap after GC invocations=458 (full 229):
 PSYoungGen      total 78336K, used 0K [0x0000000795580000, 0x000000079aa80000, 0x00000007c0000000)
  eden space 69632K, 0% used [0x0000000795580000,0x0000000795580000,0x0000000799980000)
  from space 8704K, 0% used [0x0000000799980000,0x0000000799980000,0x000000079a200000)
  to   space 8704K, 0% used [0x000000079a200000,0x000000079a200000,0x000000079aa80000)
 ParOldGen       total 175104K, used 6258K [0x0000000740000000, 0x000000074ab00000, 0x0000000795580000)
  object space 175104K, 3% used [0x0000000740000000,0x000000074061ca80,0x000000074ab00000)
 Metaspace       used 10066K, capacity 10572K, committed 10752K, reserved 1058816K
  class space    used 1151K, capacity 1252K, committed 1280K, reserved 1048576K
}

Java获取进程物理内存和虚拟内存值

import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.lang.management.MemoryUsage;
import java.math.BigInteger;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class TestXXXX {

  private static final Pattern PROCFS_STAT_FILE_FORMAT = Pattern.compile(
      "^([\\d-]+)\\s\\((.*)\\)\\s[^\\s]\\s([\\d-]+)\\s([\\d-]+)\\s" +
          "([\\d-]+)\\s([\\d-]+\\s){7}(\\d+)\\s(\\d+)\\s([\\d-]+\\s){7}(\\d+)\\s" +
          "(\\d+)(\\s[\\d-]+){15}");

  public static String getPid() {
    String name = ManagementFactory.getRuntimeMXBean().getName();
    String pid = name.split("@")[0];
    return pid;
  }

  public static void main(String[] args) throws InterruptedException, IOException {
    int[] a = new int[10240];
    // jvm-profiler 读取的文件，读取后作为Map分析
    System.out.println("===================   /proc/self/status ");
    for (String s : Files.readAllLines(Paths.get("/proc/self/status"))) {
      System.out.println(s);
    }

    // 用于测试 self 下的文件，其实只是对应pid下的文件的一个引用
    String pid = getPid();
    System.out.println("===================   /proc/" + pid + "/status ");
    for (String s : Files.readAllLines(Paths.get("/proc/" + pid + "/status"))) {
      System.out.println(s);
    }

    // hadoop-2.6.5
    // yarn / mapreduce1 ProcfsBasedProcessTree 需要读取其他进程的内存使用信息，并通过正则表达式来解析
    // vsize 虚拟内存，单位: byte，对应上面的 VmSize
    // rssmemPage 物理内存，单位PAGE，对应上面的 VmRSS， 在yarn跟踪内存使用的使用，自动 * 系统 PAGESIZE
    System.out.println("===================   /proc/" + pid + "/stat ");
    for (String s : Files.readAllLines(Paths.get("/proc/" + pid + "/stat"))) {
      Matcher m = PROCFS_STAT_FILE_FORMAT.matcher(s);
      boolean mat = m.find();
      if (mat) {
        String processName = "(" + m.group(2) + ")";
        // Set (name) (ppid) (pgrpId) (session) (utime) (stime) (vsize) (rss)
        System.out.println("name:" + processName);
        System.out.println("ppid:" + m.group(3));
        System.out.println("pgrpId:" + Integer.parseInt(m.group(4)));
        System.out.println("session:" + Integer.parseInt(m.group(5)));
        System.out.println("utime:" + Long.parseLong(m.group(7)));
        System.out.println("stime:" + new BigInteger(m.group(8)));
        System.out.println("vsize:" + Long.parseLong(m.group(10)));
        System.out.println("rssmemPage :" + Long.parseLong(m.group(11)));
        System.out.println("===================");
      }
      System.out.println(s);
    }

    // hadoop-2.6.5
    // LinuxResourceCalculatorPlugin 读取该文件，然后读取对应的内存使用信息 MemTotal 和 SwapTotal 分别为物理内存和虚拟内存的使用量
    System.out.println("===================   /proc/meminfo ");
    for (String s : Files.readAllLines(Paths.get("/proc/meminfo"))) {
      System.out.println(s);
    }

    // 输出结果和JVM 系统统计对比
    System.out.println("=================== Heap Usaging");
    Runtime runtime = Runtime.getRuntime();
    System.out.println("Used Memory:"
        + (runtime.totalMemory() - runtime.freeMemory()));
    System.out.println("Free Memory:" + runtime.freeMemory());
    System.out.println("Total Memory:" + runtime.totalMemory());
    System.out.println("Max Memory:" + runtime.maxMemory());

    // 输出结果和JVM Metrics对比
    MemoryMXBean memoryMXBean = ManagementFactory.getMemoryMXBean();

    System.out.println("\nNon Heap Usage:");
    MemoryUsage usage = memoryMXBean.getHeapMemoryUsage();
    System.out.println("INT HEAP:" + usage.getInit());
    System.out.println("MAX HEAP:" + usage.getMax());
    System.out.println("USED HEAP:" + usage.getUsed());
    System.out.println("COMMITTED HEAP:" + usage.getCommitted());

    System.out.println("\nNon Heap Usage:");
    MemoryUsage nonUsage = memoryMXBean.getNonHeapMemoryUsage();
    System.out.println("INT NON HEAP:" + nonUsage.getInit());
    System.out.println("MAX NON HEAP:" + nonUsage.getMax());
    System.out.println("USED NON HEAP:" + nonUsage.getUsed());
    System.out.println("COMMITTED NON HEAP:" + nonUsage.getCommitted());

    System.out.println("\nFull Information:");
    System.out.println("Heap Memory Usage:" + memoryMXBean.getHeapMemoryUsage());
    System.out.println("Non-Heap Memory Usage:" + memoryMXBean.getNonHeapMemoryUsage());
  }
}

输出结果:

===================   /proc/self/status
Name:	java
Umask:	0002
State:	S (sleeping)
Tgid:	20078
Ngid:	0
Pid:	20078
PPid:	8194
TracerPid:	0
Uid:	3999	3999	3999	3999
Gid:	3999	3999	3999	3999
FDSize:	256
Groups:	3999
VmPeak:	37196596 kB
VmSize:	37138988 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	   28340 kB
VmRSS:	   28340 kB
RssAnon:	   16428 kB
RssFile:	   11912 kB
RssShmem:	       0 kB
VmData:	36988292 kB
VmStk:	     132 kB
VmExe:	       4 kB
VmLib:	   16868 kB
VmPTE:	     608 kB
VmSwap:	       0 kB
Threads:	46
SigQ:	0/514997
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	2000000181005ccf
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
Seccomp:	0
Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	ffffffff
Cpus_allowed_list:	0-31
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	2
nonvoluntary_ctxt_switches:	1
===================   /proc/20078/status
Name:	java
Umask:	0002
State:	S (sleeping)
Tgid:	20078
Ngid:	0
Pid:	20078
PPid:	8194
TracerPid:	0
Uid:	3999	3999	3999	3999
Gid:	3999	3999	3999	3999
FDSize:	256
Groups:	3999
VmPeak:	37196596 kB
VmSize:	37141072 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	   28952 kB
VmRSS:	   28952 kB
RssAnon:	   16860 kB
RssFile:	   12092 kB
RssShmem:	       0 kB
VmData:	36988292 kB
VmStk:	     132 kB
VmExe:	       4 kB
VmLib:	   16904 kB
VmPTE:	     612 kB
VmSwap:	       0 kB
Threads:	46
SigQ:	0/514997
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	2000000181005ccf
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
Seccomp:	0
Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	ffffffff
Cpus_allowed_list:	0-31
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	2
nonvoluntary_ctxt_switches:	1
===================   /proc/20078/stat
name:(java)
ppid:8194
pgrpId:20078
session:1415
utime:8
stime:3
vsize:38032457728
rssmemPage :7238
===================
20078 (java) S 8194 20078 1415 34816 20078 1077944320 7714 0 0 0 8 3 0 0 20 0 46 0 1332300032 38032457728 7238 18446744073709551615 4194304 4196468 140728869431616 140728869414160 140013033029447 0 0 0 16800975 18446744073709551615 0 0 17 19 0 0 0 0 0 6293624 6294260 31469568 140728869439255 140728869439269 140728869439269 140728869441496 0
===================   /proc/meminfo
MemTotal:       131860424 kB
MemFree:        68582640 kB
MemAvailable:   105766880 kB
Buffers:         2999352 kB
Cached:         31810600 kB
SwapCached:            0 kB
Active:         40760368 kB
Inactive:       18308260 kB
Active(anon):   24259128 kB
Inactive(anon):      380 kB
Active(file):   16501240 kB
Inactive(file): 18307880 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               396 kB
Writeback:             0 kB
AnonPages:      24259440 kB
Mapped:            83296 kB
Shmem:               740 kB
Slab:            3297344 kB
SReclaimable:    3165028 kB
SUnreclaim:       132316 kB
KernelStack:       24592 kB
PageTables:        69388 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    65930212 kB
Committed_AS:   34469924 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      224452 kB
VmallocChunk:   34359504636 kB
HardwareCorrupted:     0 kB
AnonHugePages:  16392192 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      274300 kB
DirectMap2M:    15454208 kB
DirectMap1G:    120586240 kB
=================== Heap Usaging
Used Memory:42278648
Free Memory:1981997320
Total Memory:2024275968
Max Memory:28631367680

Non Heap Usage:
INT HEAP:2111832064
MAX HEAP:28631367680
USED HEAP:42278648
COMMITTED HEAP:2024275968

Non Heap Usage:
INT NON HEAP:2555904
MAX NON HEAP:-1
USED NON HEAP:7455544
COMMITTED NON HEAP:9109504

Full Information:
Heap Memory Usage:init = 2111832064(2062336K) used = 42278648(41287K) committed = 2024275968(1976832K) max = 28631367680(27960320K)
Non-Heap Memory Usage:init = 2555904(2496K) used = 7455608(7280K) committed = 9109504(8896K) max = -1(-1K)

对Jvm Heap做限制，再测试 java -Xms256m -Xmx2048m TestXXXX

===================   /proc/self/status
Name:	java
Umask:	0002
State:	S (sleeping)
Tgid:	20282
Ngid:	0
Pid:	20282
PPid:	8194
TracerPid:	0
Uid:	3999	3999	3999	3999
Gid:	3999	3999	3999	3999
FDSize:	256
Groups:	3999
VmPeak:	 6708324 kB
VmSize:	 6708320 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	   22544 kB
VmRSS:	   22544 kB
RssAnon:	   10564 kB
RssFile:	   11980 kB
RssShmem:	       0 kB
VmData:	 6557624 kB
VmStk:	     132 kB
VmExe:	       4 kB
VmLib:	   16868 kB
VmPTE:	     596 kB
VmSwap:	       0 kB
Threads:	46
SigQ:	0/514997
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	2000000181005ccf
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
Seccomp:	0
Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	ffffffff
Cpus_allowed_list:	0-31
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	2
nonvoluntary_ctxt_switches:	1
===================   /proc/20282/status
Name:	java
Umask:	0002
State:	S (sleeping)
Tgid:	20282
Ngid:	0
Pid:	20282
PPid:	8194
TracerPid:	0
Uid:	3999	3999	3999	3999
Gid:	3999	3999	3999	3999
FDSize:	256
Groups:	3999
VmPeak:	 6710408 kB
VmSize:	 6710404 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	   23172 kB
VmRSS:	   23172 kB
RssAnon:	   11040 kB
RssFile:	   12132 kB
RssShmem:	       0 kB
VmData:	 6557624 kB
VmStk:	     132 kB
VmExe:	       4 kB
VmLib:	   16904 kB
VmPTE:	     600 kB
VmSwap:	       0 kB
Threads:	46
SigQ:	0/514997
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	2000000181005ccf
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
Seccomp:	0
Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	ffffffff
Cpus_allowed_list:	0-31
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	2
nonvoluntary_ctxt_switches:	1
===================   /proc/20282/stat
name:(java)
ppid:8194
pgrpId:20282
session:1415
utime:9
stime:2
vsize:6871453696
rssmemPage :5793
===================
20282 (java) S 8194 20282 1415 34816 20282 1077944320 6248 0 0 0 9 2 0 0 20 0 46 0 1332308619 6871453696 5793 18446744073709551615 4194304 4196468 140732017816272 140732017798816 140069997956935 0 0 0 16800975 18446744073709551615 0 0 17 9 0 0 0 0 0 6293624 6294260 14270464 140732017821444 140732017821477 140732017821477 140732017823704 0
===================   /proc/meminfo
MemTotal:       131860424 kB
MemFree:        68587716 kB
MemAvailable:   105772468 kB
Buffers:         2999352 kB
Cached:         31811092 kB
SwapCached:            0 kB
Active:         40754112 kB
Inactive:       18308324 kB
Active(anon):   24252424 kB
Inactive(anon):      380 kB
Active(file):   16501688 kB
Inactive(file): 18307944 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               196 kB
Writeback:             0 kB
AnonPages:      24252032 kB
Mapped:            83300 kB
Shmem:               740 kB
Slab:            3297632 kB
SReclaimable:    3165028 kB
SUnreclaim:       132604 kB
KernelStack:       24576 kB
PageTables:        69352 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    65930212 kB
Committed_AS:   31686388 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      224452 kB
VmallocChunk:   34359504636 kB
HardwareCorrupted:     0 kB
AnonHugePages:  16392192 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      274300 kB
DirectMap2M:    15454208 kB
DirectMap1G:    120586240 kB
=================== Heap Usaging
Used Memory:4026592
Free Memory:253398816
Total Memory:257425408
Max Memory:1908932608

Non Heap Usage:
INT HEAP:268435456
MAX HEAP:1908932608
USED HEAP:4026592
COMMITTED HEAP:257425408

Non Heap Usage:
INT NON HEAP:2555904
MAX NON HEAP:-1
USED NON HEAP:7465712
COMMITTED NON HEAP:9109504

Full Information:
Heap Memory Usage:init = 268435456(262144K) used = 4026592(3932K) committed = 257425408(251392K) max = 1908932608(1864192K)
Non-Heap Memory Usage:init = 2555904(2496K) used = 7465776(7290K) committed = 9109504(8896K) max = -1(-1K)

结论：

• RSS 是比Heap使用小的，也可能比HEAP大
• VmSize 还是非常的大，即使是限制了HEAP Xmx上限