前言
怎样爬取50070页面的信息,一开始想写个爬虫,后来发现,可以通过接口调用,节省了很多麻烦
一、获取jvm使用情况信息
}[root@hadoop101 ~]# curl hadoop101:50070/jmx?qry=java.lang:type=Memory
{
"beans" : [ {
"name" : "java.lang:type=Memory",
"modelerType" : "sun.management.MemoryImpl",
"ObjectPendingFinalizationCount" : 0,
"HeapMemoryUsage" : {
"committed" : 149946368,
"init" : 130023424,
"max" : 932184064,
"used" : 111460200
},
"NonHeapMemoryUsage" : {
"committed" : 69468160,
"init" : 2555904,
"max" : -1,
"used" : 68213544
},
"Verbose" : false,
"ObjectName" : "java.lang:type=Memory"
} ]
}[root@hadoop101 ~]#
上面的单位需要转化,下面这个命令直接是MB单位
}[root@hadoop101 ~]# curl hadoop101:50070/jmx?qry=Hadoop:service=NameNode,name=JvmMetrics
{
"beans" : [ {
"name" : "Hadoop:service=NameNode,name=JvmMetrics",
"modelerType" : "JvmMetrics",
"tag.Context" : "jvm",
"tag.ProcessName" : "NameNode",
"tag.SessionId" : null,
"tag.Hostname" : "hadoop101",
"MemNonHeapUsedM" : 65.561195,
"MemNonHeapCommittedM" : 66.8125,
"MemNonHeapMaxM" : -9.536743E-7,
"MemHeapUsedM" : 123.36322,
"MemHeapCommittedM" : 153.5,
"MemHeapMaxM" : 889.0,
"MemMaxM" : 889.0,
"GcCount" : 3282,
"GcTimeMillis" : 36852,
"GcNumWarnThresholdExceeded" : 0,
"GcNumInfoThresholdExceeded" : 1,
"GcTotalExtraSleepTime" : 114272,
"ThreadsNew" : 0,
"ThreadsRunnable" : 7,
"ThreadsBlocked" : 0,
"ThreadsWaiting" : 3,
"ThreadsTimedWaiting" : 26,
"ThreadsTerminated" : 0,
"LogFatal" : 0,
"LogError" : 5,
"LogWarn" : 10,
"LogInfo" : 1302813
} ]
}
对应的页面监控信息如下
二、获取的所有信息
curl hadoop101:50070/jmx
三、获取指定信息方法
通过name参数指定,curl hadoop101:50070/jmx?qry=${name的value值}
例如:
curl hadoop101:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState
curl hadoop101:50070/jmx 查出name的值,方法上面的参数中
"name" : "Hadoop:service=NameNode,name=FSNamesystemState",
四、更多namenode的参数信息
NameNode
NameNode:50070
Hadoop:service=NameNode,name=JvmMetrics
MemHeapMaxM
MemMaxM
Hadoop:service=NameNode,name=FSNamesystem
CapacityTotal
CapacityTotalGB
CapacityRemaining
CapacityRemainingGB
TotalLoad
FilesTotal
Hadoop:service=NameNode,name=FSNamesystemState
NumLiveDataNodes
Hadoop:service=NameNode,name=NameNodeInfo
LiveNodes
java.lang:type=Runtime
StartTime
Hadoop:service=NameNode,name=FSNamesystemState
TopUserOpCounts:timestamp
参考: https://www.cnblogs.com/xinfang520/p/10740934.html
总结
通过直接调用 /jmx,访问servlet,来实现数据的获取
官网在这里写了metrix的描述,好像
https://hadoop.apache.org/docs/r2.10.1/hadoop-project-dist/hadoop-common/Metrics.html