JMX Port Monitoring for Cloudera CDH4

By default, JMX info is only accessible via the JMX JSON servet (https://issues.apache.org/jira/browse/HDFS-2083). For example, you can query the NameNode status via:

curl -i http://n1.example.com:50070/jmx

 or be more specifically:

curl -i http://n1.example.com:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo

 or use your browser as an alternative. The output of the last query is:

HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Mon, 08 Jul 2013 14:56:16 GMT
Date: Mon, 08 Jul 2013 14:56:16 GMT
Pragma: no-cache
Expires: Mon, 08 Jul 2013 14:56:16 GMT
Date: Mon, 08 Jul 2013 14:56:16 GMT
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Server: Jetty(6.1.26.cloudera.2)

{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeInfo",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    "Threads" : 65,
    "Total" : 78674993973,
    "ClusterId" : "cluster50",
    "BlockPoolId" : "BP-1455653694-192.168.1.201-1373100840050",
    "Used" : 219303936,
    "PercentUsed" : 0.2787467,
    "PercentRemaining" : 84.45627,
    "Version" : "2.0.0-cdh4.3.0, r48a9315b342ca16de92fcc5be95ae3650629155a",
    "Free" : 66445963264,
    "Safemode" : "",
    "UpgradeFinalized" : true,
    "NonDfsUsedSpace" : 12009726773,
    "BlockPoolUsedSpace" : 219303936,
    "PercentBlockPoolUsed" : 0.2787467,
    "TotalBlocks" : 228,
    "TotalFiles" : 385,
    "NumberOfMissingBlocks" : 0,
    "LiveNodes" : "{\"n3.example.com\":{\"numBlocks\":228,\"usedSpace\":73142272,\"lastContact\":2,\"capacity\":26224997991,\"nonDfsUsedSpace\":3542431335,\"adminState\":\"In Service\"},\"n2.example.com\":{\"numBlocks\":223,\"usedSpace\":73080832,\"lastContact\":1,\"capacity\":26224997991,\"nonDfsUsedSpace\":3546506855,\"adminState\":\"In Service\"},\"n1.example.com\":{\"numBlocks\":223,\"usedSpace\":73080832,\"lastContact\":0,\"capacity\":26224997991,\"nonDfsUsedSpace\":4920788583,\"adminState\":\"In Service\"}}",
    "DeadNodes" : "{}",
    "DecomNodes" : "{}",
    "NameDirStatuses" : "{\"failed\":{},\"active\":{\"/dfs/nn\":\"IMAGE_AND_EDITS\"}}"
  } ]

 

 

You can also configure hadoop-env.sh (usually located in /etc/hadoop/..., for example, the mapreduce1 configure file path is /etc/hadoop/conf.cloudera.hdfs1) in CDH4, update HADOOP_NAMENODE_OPTS with 

-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.password.file=$HADOOP_CONF_DIR/jmxremote.password
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8004

 Then connect to the JMX server at port 8004 with JConsole, VisualVM or use the built-in JMXGet tool as well.

[root@n1 conf.cloudera.hdfs1]# hadoop org.apache.hadoop.hdfs.tools.JMXGet --help
usage: jmxget options are:
 -help                           print help
 -localVM <VM's connector url>   connect to the VM on the same machine;
                                 use:
                                 jstat -J-Djstat.showUnsupported=true
                                 -snap <vmpid> | grep
                                 sun.management.JMXConnectorServer.address
                                 to find the url
 -port <mbean server port>       specify mbean server port, if missing -
                                 it will try to connect to MBean Server in
                                 the same VM
 -server <mbean server>          specify mbean server (localhost by
                                 default)
 -service <NameNode|DataNode>    specify jmx service (NameNode by default)

 

Above is the configuration of NameNode, you can configure the JMX monitoring to SecondaryNameNode, DataNode, JobTracker, TaskTracker just like the NameNode. See this blog post.