What is Hadoop Metrics2?

转载 2015年11月19日 21:23:01

source:http://blog.cloudera.com/blog/2012/10/what-is-hadoop-metrics2/


Metrics are collections of information about Hadoop daemons, events and measurements; for example, data nodes collect metrics such as the number of blocks replicated, number of read requests from clients, and so on. For that reason, metrics are an invaluable resource for monitoring Apache Hadoop services and an indispensable tool for debugging system problems. 

This blog post focuses on the features and use of the Metrics2 system for Hadoop, which allows multiple metrics output plugins to be used in parallel, supports dynamic reconfiguration of metrics plugins, provides metrics filtering, and allows all metrics to be exported via JMX.

Metrics vs. MapReduce Counters

When speaking about metrics, a question about their relationship to MapReduce counters usually arises. This differences can be described in two ways: First, Hadoop daemons and services are generally the scope for metrics, whereas MapReduce applications are the scope for MapReduce counters (which are collected for MapReduce tasks and aggregated for the whole job). Second, whereas Hadoop administrators are the main audience for metrics, MapReduce users are the audience for MapReduce counters.

Contexts and Prefixes

For organizational purposes metrics are grouped into named contexts – e.g., jvm for java virtual machine metrics or dfs for the distributed file system metric. There are different sets of contexts supported by Hadoop-1 and Hadoop-2; the table below highlights the ones supported for each of them.  

Branch-1

Branch-2

– jvm
– rpc
– rpcdetailed
– metricssystem
– mapred
– dfs
– ugi
– yarn
– jvm
– rpc
– rpcdetailed
– metricssystem
– mapred
– dfs
– ugi

A Hadoop daemon collects metrics in several contexts. For example, data nodes collect metrics for the “dfs”, “rpc” and “jvm” contexts. The daemons that collect different metrics in Hadoop (for Hadoop-1 and Hadoop-2) are listed below:

Branch-1 Daemons/Prefixes Branch-2 Daemons/Prefixes

– namenode
– datanode
– jobtracker
– tasktracker
– maptask
– reducetask

 

– namenode
– secondarynamenode
– datanode
– resourcemanager
– nodemanager
– mrappmaster
– maptask
– reducetask

System Design

The Metrics2 framework is designed to collect and dispatch per-process metrics to monitor the overall status of the Hadoop system. Producers register the metrics sources with the metrics system, while consumers register the sinks. The framework marshals metrics from sources to sinks based on (per source/sink) configuration options. This design is depicted below.

 

Here is an example class implementing the MetricsSource:

The “MyMetric” in the listing above could be, for example, the number of open connections for a specific server.

Here is an example class implementing the MetricsSink:

To use the Metric2s framework, the system needs to be initialized and sources and sinks registered. Here is an example initialization:

Configuration and Filtering

The Metrics2 framework uses the PropertiesConfiguration from the apache commons configuration library.

Sinks are specified in a configuration file (e.g., “hadoop-metrics2-test.properties”), as:

The configuration syntax is:

In the previous example, test is the prefix and mysink0 is an instance name. DefaultMetricsSystem would try to load hadoop-metrics2-[prefix].properties first, and if not found, try the default hadoop-metrics2.properties in the class path. Note, the [instance] is an arbitrary name to uniquely identify a particular sink instance. The asterisk (*) can be used to specify default options.

Here is an example with inline comments to identify the different configuration sections:

Here is an example set of NodeManager metrics that are dumped into the NodeManager sink file:

Each line starts with a time followed by the context and metrics name and the corresponding value for each metric.

Filtering

By default, filtering can be done by source, context, record and metrics. More discussion of different filtering strategies can be found in the Javadoc and wiki.

Example:

Conclusion

The Metrics2 system for Hadoop provides a gold mine of real-time and historical data that help monitor and debug problems associated with the Hadoop services and jobs. 


ganglia在hadoop中的配置:hadoop-metrics2.properties 以及各监控项含义

http://blog.csdn.net/ljq32/article/details/37662665 ganglia3.6.0  hadoop2.0.0-cdh4.2.0 使用 hadoo...
  • beyondlpf
  • beyondlpf
  • 2015年06月18日 18:53
  • 2073

Coursea Hadoop课堂笔记——Lesson 1: Big Data Hadoop Stack

1. Hadoop move computation to data 2. scalability、reliability、 3. Apache框架基本模块: Hadoop Common HDFS ...
  • ztf312
  • ztf312
  • 2016年01月15日 21:36
  • 2139

Hadoop Metrics体系分析之二:Hadoop Metrics实现

我们如何才能获知一个软件系统的运行状况?如何才能将软件的运行数据暴露给用户查看?通过hadoop的metrics框架就能做到这一点。它可以帮助我们计算数据,收集数据,发送数据,这一切仅需要我们建立几个...
  • huoyunshen88
  • huoyunshen88
  • 2014年02月07日 20:35
  • 2700

What Hadoop is Not

We see a lot of emails where people hear about Hadoop, and think it will be the silver bullet to sol...
  • mozart_cai
  • mozart_cai
  • 2013年01月11日 11:55
  • 325

what is hadoop?

  • gz153016
  • gz153016
  • 2016年02月01日 22:51
  • 188

ganglia在hadoop中的配置:hadoop-metrics2.properties

ganglia3.6.0  hadoop2.0.0-cdh4.2.0 使用 hadoop-metrics2.properties, 在文件中添加以下内容,...
  • ljq32
  • ljq32
  • 2014年07月10日 19:05
  • 2949

What Why How Hadoop

What 其实每学习一门新东西的时候,我们都在问自己,这东西是什么,为什么要学习,它能被用来做什么。不知道你有没有注意到:网上一些国外著名大学的网络课程,往往第一个视频里,无论是专家还是教授都在...
  • thinking4java
  • thinking4java
  • 2012年03月12日 20:24
  • 454

Alex 的 Hadoop 菜鸟教程: 第21课 不只是在HBase中用SQL:Phoenix

什么是Phoenix? Phoenix的团队用了一句话概括Phoenix:"We put the SQL back in NoSQL" 意思是:我们把SQL又放回NoSQL去了!这边说的NoSQL专指...
  • nsrainbow
  • nsrainbow
  • 2015年03月06日 10:37
  • 7269

What Hadoop is good at

转自: http://horicky.blogspot.com/2009/11/what-hadoop-is-good-at.html Hadoop is getting more ...
  • cloudeagle_bupt
  • cloudeagle_bupt
  • 2013年07月14日 14:49
  • 553

What's Hadoop

What's Hadoop? 一、hadoop起源 hadoop的发起者是Doug Cutting,同时他也是Lucene、Nutch的发起者。此人伟大之处在于一生致力于开源产品的研发,“把高深莫...
  • Zhanggs
  • Zhanggs
  • 2014年04月18日 17:22
  • 489
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:What is Hadoop Metrics2?
举报原因:
原因补充:

(最多只允许输入30个字)