mapreduce counter

最新推荐文章于 2024-06-27 10:25:41 发布

kingjo002

最新推荐文章于 2024-06-27 10:25:41 发布

阅读量1k

点赞数

本文链接：https://blog.csdn.net/kingjo002/article/details/20636631

版权

http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-counter/

http://stackoverflow.com/questions/8616041/how-do-i-get-counters-in-hadoop

http://blog.csdn.net/posa88/article/details/7904720

mark

If you are submitting your job like this:

  Configuration conf = new Configuration();
  Job job = new Job(conf);

  job.waitForCompletion(true);

And it has finished (you can call this even when its running, but the results won't be final then, because the job hasn't completed yet.), you can grab the counters by using:

long counter = job.getCounters().findCounter(ExplorationReducer.UpdateCounter.UPDATED)
    .getValue();

This is the name of the enum counter I used in my job:

ExplorationReducer.UpdateCounter.UPDATED

If you want all counter you have to traverse the backing structure behind the Counters object. There is an iterator for it.

An Example of Hadoop MapReduce Counter

Posted: March 15, 2011| Author: Hyunsik Choi | Filed under:FOSS | Tags: counter, hadoop, mapreduce |2 Comments

MapReduce Counter

Hadoop MapReduce Counter provides a way to measure the progress or the number of operations that occur within MapReduce programs. Basically, MapReduce framework provides a number of built-in counters to measure basic I/O operations, such as FILE_BYTES_READ/WRITTEN and Map/Combine/Reduce input/output records. These counters are very useful especially when you evaluate some MapReduce programs. Besides, the MapReduce Counter allows users to employ your own counters. Since MapReduce Counters are automatically aggregated over Map and Reduce phases, it is one of the easiest way to investigate internal behaviors of MapReduce programs. In this post, I’m going to introduce how to use your own MapReduce Counter. The example sources described in this post are based on Hadoop 0.21 API.

Incrementing your counter

For your own MapReduce counter, you first define a enum type as follow:

 
        public 
         static 
        enum 
        MATCH_COUNTER { 
       
        INCOMING_GRAPHS, 
       
        PRUNING_BY_NCV, 
       
        PRUNING_BY_COUNT, 
       
        PRUNING_BY_ISO, 
       
        ISOMORPHIC 
       
        };

And then, when you want to increment your own counter, you should call the incrementmethod as follows:

 
        context.getCounter(MATCH_COUNTER.INCOMING_GRAPHS).increment( 
        1 
        );

You can access context instance within setup, cleanup,map, and reduce method in Mapper or Reducer class. You can get a desired counter via calling context.getCounter method with some enum value.

Finding your counter

You can get some Counters from a finished job as follows:

 
        Configuration conf = 
        new 
        Configuration(); 
       
        Cluster cluster =  
        new 
        Cluster(conf); 
       
        Job job = Job.getInstance(cluster,conf); 
       
        result = job.waitForCompletion( 
        true 
        ); 
       
        ... 
       
        Counters counters = job.getCounters();

The instance of Counters class contains all of the counters obtained from a job. So, when you want to get your own counter, you should callfindCounter method with a enum type as follows:

 
        Counter c1 = counters.findCounter(MATCH_COUNTER.INCOMING_GRAPHS); 
       
        System.out.println(c1.getDisplayName()+ 
        ":" 
        +c1.getValue());

The below example shows how to get built-in counter groups that Hadoop provides basically.

 
      for 
       (CounterGroup group : counters) { 
     
      System.out.println( 
      "* Counter Group: " 
      + group.getDisplayName() +  
      " (" 
      + group.getName() +  
      ")" 
      ); 
     
      System.out.println( 
      "  number of counters in this group: " 
      + group.size()); 
     
      for 
      (Counter counter : group) { 
     
      System.out.println( 
      "  - " 
      + counter.getDisplayName() +  
      ": " 
      + counter.getName() +  
      ": " 
      +counter.getValue()); 
     
      } 
     
      }