小文件治理之hadoop集群监控： hdfs jvm数据获取

---

前言

怎样爬取50070页面的信息，一开始想写个爬虫，后来发现，可以通过接口调用，节省了很多麻烦

---

一、获取jvm使用情况信息

}[root@hadoop101 ~]# curl hadoop101:50070/jmx?qry=java.lang:type=Memory
{
  "beans" : [ {
    "name" : "java.lang:type=Memory",
    "modelerType" : "sun.management.MemoryImpl",
    "ObjectPendingFinalizationCount" : 0,
    "HeapMemoryUsage" : {
      "committed" : 149946368,
      "init" : 130023424,
      "max" : 932184064,
      "used" : 111460200
    },
    "NonHeapMemoryUsage" : {
      "committed" : 69468160,
      "init" : 2555904,
      "max" : -1,
      "used" : 68213544
    },
    "Verbose" : false,
    "ObjectName" : "java.lang:type=Memory"
  } ]
}[root@hadoop101 ~]# 

上面的单位需要转化，下面这个命令直接是MB单位

}[root@hadoop101 ~]#  curl hadoop101:50070/jmx?qry=Hadoop:service=NameNode,name=JvmMetrics
{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=JvmMetrics",
    "modelerType" : "JvmMetrics",
    "tag.Context" : "jvm",
    "tag.ProcessName" : "NameNode",
    "tag.SessionId" : null,
    "tag.Hostname" : "hadoop101",
    "MemNonHeapUsedM" : 65.561195,
    "MemNonHeapCommittedM" : 66.8125,
    "MemNonHeapMaxM" : -9.536743E-7,
    "MemHeapUsedM" : 123.36322,
    "MemHeapCommittedM" : 153.5,
    "MemHeapMaxM" : 889.0,
    "MemMaxM" : 889.0,
    "GcCount" : 3282,
    "GcTimeMillis" : 36852,
    "GcNumWarnThresholdExceeded" : 0,
    "GcNumInfoThresholdExceeded" : 1,
    "GcTotalExtraSleepTime" : 114272,
    "ThreadsNew" : 0,
    "ThreadsRunnable" : 7,
    "ThreadsBlocked" : 0,
    "ThreadsWaiting" : 3,
    "ThreadsTimedWaiting" : 26,
    "ThreadsTerminated" : 0,
    "LogFatal" : 0,
    "LogError" : 5,
    "LogWarn" : 10,
    "LogInfo" : 1302813
  } ]
}

对应的页面监控信息如下

二、获取的所有信息

curl hadoop101:50070/jmx

三、获取指定信息方法

通过name参数指定，curl hadoop101:50070/jmx?qry=${name的value值}
例如：
curl hadoop101:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState

curl hadoop101:50070/jmx 查出name的值，方法上面的参数中

    "name" : "Hadoop:service=NameNode,name=FSNamesystemState",

四、更多namenode的参数信息

NameNode
NameNode:50070
Hadoop:service=NameNode,name=JvmMetrics
MemHeapMaxM
MemMaxM
Hadoop:service=NameNode,name=FSNamesystem
CapacityTotal
CapacityTotalGB
CapacityRemaining
CapacityRemainingGB
TotalLoad
FilesTotal
Hadoop:service=NameNode,name=FSNamesystemState
NumLiveDataNodes
Hadoop:service=NameNode,name=NameNodeInfo
LiveNodes
java.lang:type=Runtime
StartTime
Hadoop:service=NameNode,name=FSNamesystemState
TopUserOpCounts:timestamp

---

参考： https://www.cnblogs.com/xinfang520/p/10740934.html

总结

通过直接调用 /jmx，访问servlet，来实现数据的获取

官网在这里写了metrix的描述，好像
https://hadoop.apache.org/docs/r2.10.1/hadoop-project-dist/hadoop-common/Metrics.html