Hadoop Metrics研究—1

一、metrics介绍

是用来做服务统计、监控的，例如qps，每个请求的响应时间，待处理的队列长度等。

目前最为流行的 metrics 库是来自 Coda Hale 的 dropwizard/metrics，该库被广泛地应用于各个知名的开源项目中。例如 Hadoop，Kafka，Spark，JStorm 中。

Maven 配置

<dependencies>
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>${metrics.version}</version>
</dependency>
</dependencies>

Metric Registries

MetricRegistry类是Metrics的核心，它是存放应用中所有metrics的容器。也是我们使用 Metrics 库的起点。

MetricRegistry registry = new MetricRegistry();
MetricRegistry.name(Queue.class, "requests", "size")

Metrics 数据获取接口

etrics-core中主要实现了四种 reporter：

• JMX
• console
•  SLF4J 
•  CSV

五种 Metrics 类型

Gauges

最简单的度量指标，只有一个简单的返回值，例如，我们想衡量一个待处理队列中任务的个数，代码如下：

public class GaugeTest {
public static Queue<String> q = new LinkedList<String>();
public static void main(String[] args) throws InterruptedException {
MetricRegistry registry = new MetricRegistry();
ConsoleReporter reporter = ConsoleReporter.forRegistry(registry).build();
reporter.start(1, TimeUnit.SECONDS);
registry.register(MetricRegistry.name(GaugeTest.class, "queue", "size"),
new Gauge<Integer>() {
public Integer getValue() {
return q.size();
}
});
while(true){
Thread.sleep(1000);
q.add("Job-xxx");
}
}
}

-- Gauges ------------------------------------------------
com.alibaba.wuchong.metrics.GaugeTest.queue.size
value = 6

Counters

Counter 就是计数器，Counter 只是用 Gauge 封装了 AtomicLong 。

public class CounterTest {
public static Queue<String> q = new LinkedBlockingQueue<String>();
public static Counter pendingJobs;
public static Random random = new Random();
public static void addJob(String job) {
pendingJobs.inc();
q.offer(job);
}
public static String takeJob() {
pendingJobs.dec();
return q.poll();
}
public static void main(String[] args) throws InterruptedException {
MetricRegistry registry = new MetricRegistry();
ConsoleReporter reporter = ConsoleReporter.forRegistry(registry).build();
reporter.start(1, TimeUnit.SECONDS);
pendingJobs = registry.counter(MetricRegistry.name(Queue.class,"pending-jobs","size"));
int num = 1;
while(true){
Thread.sleep(200);
if (random.nextDouble() > 0.7){
String job = takeJob();
System.out.println("take job : "+job);
}else{
String job = "Job-"+num;
addJob(job);
System.out.println("add job : "+job);
}
num++;
}
}
}

add job : Job-15
add job : Job-16
take job : Job-8
take job : Job-10
add job : Job-19
15-8-1 16:11:31 ============================================
-- Counters ----------------------------------------------
java.util.Queue.pending-jobs.size
count = 5

Meters

Meter度量一系列事件发生的速率(rate)，例如TPS。Meters会统计最近1分钟，5分钟，15分钟，还有全部时间的速率

public class MeterTest {
public static Random random = new Random();
public static void request(Meter meter){
System.out.println("request");
meter.mark();
}
public static void request(Meter meter, int n){
while(n > 0){
request(meter);
n--;
}
}
public static void main(String[] args) throws InterruptedException {
MetricRegistry registry = new MetricRegistry();
ConsoleReporter reporter = ConsoleReporter.forRegistry(registry).build();
reporter.start(1, TimeUnit.SECONDS);
Meter meterTps = registry.meter(MetricRegistry.name(MeterTest.class,"request","tps"));
while(true){
request(meterTps,random.nextInt(5));
Thread.sleep(1000);
}
}
}

request
15-8-1 16:23:25 ============================================
-- Meters ------------------------------------------------
com.alibaba.wuchong.metrics.MeterTest.request.tps
count = 134
mean rate = 2.13 events/second
1-minute rate = 2.52 events/second
5-minute rate = 3.16 events/second
15-minute rate = 3.32 events/second

Histograms

Histogram统计数据的分布情况。比如最小值，最大值，中间值，还有中位数，和 百分比的值(percentiles)，比如request的大小的分布：

public class HistogramTest {
public static Random random = new Random();
public static void main(String[] args) throws InterruptedException {
MetricRegistry registry = new MetricRegistry();
ConsoleReporter reporter = ConsoleReporter.forRegistry(registry).build();
reporter.start(1, TimeUnit.SECONDS);
Histogram histogram = new Histogram(new ExponentiallyDecayingReservoir());
registry.register(MetricRegistry.name(HistogramTest.class, "request", "histogram"), histogram);
while(true){
Thread.sleep(1000);
histogram.update(random.nextInt(100000));}}
}

-- Histograms --------------------------------------------
java.util.Queue.queue.histogram
count = 56
min = 1122
max = 99650
mean = 48735.12
stddev = 28609.02
median = 49493.00
75% <= 72323.00
95% <= 90773.00
98% <= 94011.00
99% <= 99650.00
99.9% <= 99650.00

Timers

Timer其实是 Histogram 和 Meter 的结合， histogram 某部分代码/调用的耗时， meter统计TPS。

public class TimerTest {
public static Random random = new Random();
public static void main(String[] args) throws InterruptedException {
MetricRegistry registry = new MetricRegistry();
ConsoleReporter reporter = ConsoleReporter.forRegistry(registry).build();
reporter.start(1, TimeUnit.SECONDS);
Timer timer = registry.timer(MetricRegistry.name(TimerTest.class,"get-latency"));
Timer.Context ctx;
while(true){
ctx = timer.time();
Thread.sleep(random.nextInt(1000));
ctx.stop();}}
}
-- Timers ------------------------------------------------
com.alibaba.wuchong.metrics.TimerTest.get-latency
count = 38
mean rate = 1.90 calls/second
1-minute rate = 1.66 calls/second
5-minute rate = 1.61 calls/second
15-minute rate = 1.60 calls/second
min = 13.90 milliseconds
max = 988.71 milliseconds
mean = 519.21 milliseconds
stddev = 286.23 milliseconds
median = 553.84 milliseconds
75% <= 763.64 milliseconds
95% <= 943.27 milliseconds
98% <= 988.71 milliseconds
99% <= 988.71 milliseconds
99.9% <= 988.71 milliseconds

Metrics for Spring

这个库为Spring增加了Metric库， 提供基于注解方式的metrics类库使用。

• 可以使用注解创建metric和代理类。 @Timed, @Metered, @ExceptionMetered, @Counted
• 为@Metric注解的字段自动装配

• <dependency>
      <groupId>com.ryantenney.metrics</groupId>
      <artifactId>metrics-spring</artifactId>
      <version>3.0.1</version>
  </dependency>

  @InterfaceAudience.Private
  @Metrics(context="yarn")
  public class ClusterMetrics {
    
    private static AtomicBoolean isInitialized = new AtomicBoolean(false);
    
    @Metric("# of active NMs") MutableGaugeInt numActiveNMs;
    @Metric("# of decommissioned NMs") MutableGaugeInt numDecommissionedNMs;
    @Metric("# of lost NMs") MutableGaugeInt numLostNMs;
    @Metric("# of unhealthy NMs") MutableGaugeInt numUnhealthyNMs;
    @Metric("# of Rebooted NMs") MutableGaugeInt numRebootedNMs;
    @Metric("AM container launch delay") MutableRate aMLaunchDelay;
    @Metric("AM register delay") MutableRate aMRegisterDelay;
  
    private static final MetricsInfo RECORD_INFO = info("ClusterMetrics",
    "Metrics for the Yarn Cluster");
    
    private static volatile ClusterMetrics INSTANCE = null;
    private static MetricsRegistry registry;
    
    public static ClusterMetrics getMetrics() {
      if(!isInitialized.get()){
        synchronized (ClusterMetrics.class) {
          if(INSTANCE == null){
            INSTANCE = new ClusterMetrics();
            registerMetrics();
            isInitialized.set(true);
          }
        }
      }
      return INSTANCE;
    }

  hadoop源码作为一个Spring项目，MetricRegistry常常会被声明成一个Bean，然后在有需要的地方进行注入，最后就可以达到只通过一个MetricRegistry，就能够收集到所有指标的效果。另外需要收集的各种指标，会被定义为5种metircs的子类，这样我们就可以通过registry.getCounters/getHistograms/getGauges来获取我们所有注册过的metrics指标了。