1、计数器简介
在许多情况下,一个用户需要了解待分析的数据,尽管这并非所要执行的分析任务 的核心内容。以统计数据集中无效记录数目的任务为例,如果发现无效记录的比例 相当高,那么就需要认真思考为何存在如此多无效记录。是所采用的检测程序存在 缺陷,还是数据集质量确实很低,包含大量无效记录?如果确定是数据集的质量问 题,则可能需要扩大数据集的规模,以增大有效记录的比例,从而进行有意义的 分析。
计数器是一种收集作业统计信息的有效手段,用于质量控制或应用级统计。计数器 还可辅助诊断系统故障。如果需要将日志信息传输到map或reduce任务,更好的 方法通常是尝试传输计数器值以监测某一特定事件是否发生。对于大型分布式作业 而言,使用计数器更为方便。首先,获取计数器值比输出日志更方便,其次,根据 计数器值统计特定事件的发生次数要比分析一堆日志文件容易得多。
2、内置计数器
Hadoop为每个作业维护若干内置计数器, 以描述该作业的各项指标。例如,某些计数器记录已处理的字节数和记录数,使用户可监控已处理的输入数据量和已产生的输出数据量,并以此对 job 做适当的优化。
01 | 14/06/08 15:13:35 INFO mapreduce.Job: Counters: 46 |
03 | FILE: Number of bytes read =159 |
04 | FILE: Number of bytes written=159447 |
05 | FILE: Number of read operations=0 |
06 | FILE: Number of large read operations=0 |
07 | FILE: Number of write operations=0 |
08 | HDFS: Number of bytes read =198 |
09 | HDFS: Number of bytes written=35 |
10 | HDFS: Number of read operations=6 |
11 | HDFS: Number of large read operations=0 |
12 | HDFS: Number of write operations=2 |
15 | Launched reduce tasks=1 |
16 | Rack- local map tasks=1 |
17 | Total time spent by all maps in occupied slots (ms)=3896 |
18 | Total time spent by all reduces in occupied slots (ms)=9006 |
23 | Map output materialized bytes=159 |
25 | Combine input records=0 |
26 | Combine output records=0 |
28 | Reduce shuffle bytes=159 |
29 | Reduce input records=12 |
30 | Reduce output records=4 |
35 | GC time elapsed (ms)=13 |
36 | CPU time spent (ms)=3830 |
37 | Physical memory (bytes) snapshot=537718784 |
38 | Virtual memory (bytes) snapshot=7365263360 |
39 | Total committed heap usage (bytes)=2022309888 |
47 | File Input Format Counters |
49 | File Output Format Counters |
计数器由其关联任务维护,并定期传到tasktracker,再由tasktracker传给 jobtracker.因此,计数器能够被全局地聚集。详见第 hadoop 权威指南第170页的“进度和状态的更新”小节。与其他计数器(包括用户定义的计数器)不同,内置的作业计数器实际上 由jobtracker维护,不必在整个网络中发送。
一个任务的计数器值每次都是完整传输的,而非自上次传输之后再继续数未完成的传输,以避免由于消息丢失而引发的错误。另外,如果一个任务在作业执行期间失 败,则相关计数器值会减小。仅当一个作业执行成功之后,计数器的值才是完整可 靠的。
3、用户定义的Java计数器
MapReduce允许用户编写程序来定义计数器,计数器的值可在mapper或reducer 中增加。多个计数器由一个Java枚举(enum)类型来定义,以便对计数器分组。一 个作业可以定义的枚举类型数量不限,各个枚举类型所包含的字段数量也不限。枚 举类型的名称即为组的名称,枚举类型的字段就是计数器名称。计数器是全局的。 换言之,MapReduce框架将跨所有map和reduce聚集这些计数器,并在作业结束 时产生一个最终结果。
Notice1:需要说明的是,不同的 hadoop 版本定义的方式会有些许差异。
(1)在0.20.x版本中使用counter很简单,直接定义即可,如无此counter,hadoop会自动添加此counter.
1 | Counter ct = context.getCounter( "INPUT_WORDS" , "count" ); |
(2)在0.19.x版本中,需要定义enum
1 | enum MyCounter {INPUT_WORDS }; |
2 | reporter.incrCounter(MyCounter.INPUT_WORDS, 1 ); |
3 | RunningJob job = JobClient.runJob(conf); |
4 | Counters c = job.getCounters(); |
5 | long cnt = c.getCounter(MyCounter.INPUT_WORDS); |
Notice2:使用计数器需要清楚的是它们都存储在jobTracker的内存里。Mapper/Reducer 任务序列化它们,连同更新状态被发送。为了运行正常且jobTracker不会出问题,计数器的数量应该在10-100个,计数器不仅仅只用来聚合MapReduce job的统计值。新版本的hadoop限制了计数器的数量,以防给jobTracker带来损害。你最不想看到的事情就是由于定义上百个计数器而使jobTracker宕机。
下面咱们来看一个计数器的实例(以下代码请运行在 0.20.1 版本以上):
3.1 测试数据:
1 | hello world 2013 mapreduce |
2 | hello world 2013 mapreduce |
3 | hello world 2013 mapreduce |
3.2 代码:
002 | * Project Name:CDHJobs |
003 | * File Name:MapredCounter.java |
005 | * Date:2014-6-8下午2:12:48 |
006 | * Copyright (c) 2014, decli#qq.com All Rights Reserved. |
012 | import java.io.IOException; |
013 | import java.util.StringTokenizer; |
015 | import org.apache.commons.lang3.StringUtils; |
016 | import org.apache.hadoop.conf.Configuration; |
017 | import org.apache.hadoop.fs.Path; |
018 | import org.apache.hadoop.io.IntWritable; |
019 | import org.apache.hadoop.io.Text; |
020 | import org.apache.hadoop.mapreduce.Counter; |
021 | import org.apache.hadoop.mapreduce.CounterGroup; |
022 | import org.apache.hadoop.mapreduce.Counters; |
023 | import org.apache.hadoop.mapreduce.Job; |
024 | import org.apache.hadoop.mapreduce.Mapper; |
025 | import org.apache.hadoop.mapreduce.Reducer; |
026 | import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; |
027 | import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; |
029 | public class WordCountWithCounter { |
031 | static enum WordsNature { |
032 | STARTS_WITH_DIGIT, STARTS_WITH_LETTER, ALL |
036 | * The map class of WordCount. |
038 | public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable> { |
040 | private final static IntWritable one = new IntWritable( 1 ); |
041 | private Text word = new Text(); |
043 | public void map(Object key, Text value, Context context) throws IOException, InterruptedException { |
044 | StringTokenizer itr = new StringTokenizer(value.toString()); |
045 | while (itr.hasMoreTokens()) { |
046 | word.set(itr.nextToken()); |
047 | context.write(word, one); |
053 | * The reducer class of WordCount |
055 | public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, IntWritable> { |
056 | public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, |
057 | InterruptedException { |
060 | String token = key.toString(); |
061 | if (StringUtils.isNumeric(token)) { |
062 | context.getCounter(WordsNature.STARTS_WITH_DIGIT).increment( 1 ); |
063 | } else if (StringUtils.isAlpha(token)) { |
064 | context.getCounter(WordsNature.STARTS_WITH_LETTER).increment( 1 ); |
066 | context.getCounter(WordsNature.ALL).increment( 1 ); |
068 | for (IntWritable value : values) { |
071 | context.write(key, new IntWritable(sum)); |
076 | * The main entry point. |
078 | public static void main(String[] args) throws Exception { |
079 | Configuration conf = new Configuration(); |
080 | Job job = new Job(conf, "WordCountWithCounter" ); |
081 | job.setJarByClass(WordCountWithCounter. class ); |
082 | job.setMapperClass(TokenCounterMapper. class ); |
083 | job.setReducerClass(TokenCounterReducer. class ); |
084 | job.setOutputKeyClass(Text. class ); |
085 | job.setOutputValueClass(IntWritable. class ); |
086 | FileInputFormat.addInputPath(job, new Path( "/tmp/dsap/rawdata/june/a.txt" )); |
087 | FileOutputFormat.setOutputPath(job, new Path( "/tmp/dsap/rawdata/june/a_result" )); |
088 | int exitCode = job.waitForCompletion( true ) ? 0 : 1 ; |
090 | Counters counters = job.getCounters(); |
091 | Counter c1 = counters.findCounter(WordsNature.STARTS_WITH_DIGIT); |
092 | System.out.println( "-------------->>>>: " + c1.getDisplayName() + ": " + c1.getValue()); |
095 | for (CounterGroup group : counters) { |
096 | System.out.println( "==========================================================" ); |
097 | System.out.println( "* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")" ); |
098 | System.out.println( " number of counters in this group: " + group.size()); |
099 | for (Counter counter : group) { |
100 | System.out.println( " ++++ " + counter.getDisplayName() + ": " + counter.getName() + ": " |
101 | + counter.getValue()); |
104 | System.exit(exitCode); |
3.3 结果与计数器详解
运行结果下面会一并给出。Counter有"组group"的概念,用于表示逻辑上相同范围的所有数值。MapReduce job提供的默认Counter分为7个组,下面逐一介绍。这里也拿上面的测试数据来做详细比对,我将会针对具体的计数器,挑选一些主要的简述一下。
001 | ... 前面省略 job 运行信息 xx 字 ... |
005 | -------------->>>>: STARTS_WITH_DIGIT: 1 |
006 | ========================================================== |
008 | * Counter Group: File System Counters (org.apache.hadoop.mapreduce.FileSystemCounter) |
009 | number of counters in this group: 10 |
011 | ++++ FILE: Number of bytes read : FILE_BYTES_READ: 159 |
013 | ++++ FILE: Number of bytes written: FILE_BYTES_WRITTEN: 159447 |
014 | ++++ FILE: Number of read operations: FILE_READ_OPS: 0 |
015 | ++++ FILE: Number of large read operations: FILE_LARGE_READ_OPS: 0 |
016 | ++++ FILE: Number of write operations: FILE_WRITE_OPS: 0 |
018 | ++++ HDFS: Number of bytes read : HDFS_BYTES_READ: 198 |
020 | ++++ HDFS: Number of bytes written: HDFS_BYTES_WRITTEN: 35 |
021 | ++++ HDFS: Number of read operations: HDFS_READ_OPS: 6 |
022 | ++++ HDFS: Number of large read operations: HDFS_LARGE_READ_OPS: 0 |
023 | ++++ HDFS: Number of write operations: HDFS_WRITE_OPS: 2 |
024 | ========================================================== |
026 | * Counter Group: Job Counters (org.apache.hadoop.mapreduce.JobCounter) |
027 | number of counters in this group: 5 |
029 | ++++ Data- local map tasks |
031 | ++++ FALLOW_SLOTS_MILLIS_MAPS/REDUCES |
033 | ++++ SLOTS_MILLIS_MAPS/REDUCES |
035 | ++++ Launched map tasks: TOTAL_LAUNCHED_MAPS: 1 |
037 | ++++ Launched reduce tasks: TOTAL_LAUNCHED_REDUCES: 1 |
038 | ++++ Rack- local map tasks: RACK_LOCAL_MAPS: 1 |
039 | ++++ Total time spent by all maps in occupied slots (ms): SLOTS_MILLIS_MAPS: 3896 |
040 | ++++ Total time spent by all reduces in occupied slots (ms): SLOTS_MILLIS_REDUCES: 9006 |
041 | ========================================================== |
043 | * Counter Group: Map-Reduce Framework (org.apache.hadoop.mapreduce.TaskCounter) |
044 | number of counters in this group: 20 |
046 | ++++ Map input records: MAP_INPUT_RECORDS: 3 |
048 | ++++ Map output records: MAP_OUTPUT_RECORDS: 12 |
050 | ++++ Map output bytes: MAP_OUTPUT_BYTES: 129 |
051 | ++++ Map output materialized bytes: MAP_OUTPUT_MATERIALIZED_BYTES: 159 |
053 | ++++ Input split bytes: SPLIT_RAW_BYTES: 117 |
055 | ++++ Combine input records: COMBINE_INPUT_RECORDS: 0 |
057 | ++++ Combine output records: COMBINE_OUTPUT_RECORDS: 0 |
059 | ++++ Reduce input groups : REDUCE_INPUT_GROUPS: 4 |
061 | ++++ Reduce shuffle bytes: REDUCE_SHUFFLE_BYTES: 159 |
063 | ++++ Reduce input records: REDUCE_INPUT_RECORDS: 12 |
065 | ++++ Reduce output records: REDUCE_OUTPUT_RECORDS: 4 |
067 | ++++ Spilled Records: SPILLED_RECORDS: 24 |
069 | ++++ Shuffled Maps : SHUFFLED_MAPS: 1 |
071 | ++++ Failed Shuffles: FAILED_SHUFFLE: 0 |
073 | ++++ Merged Map outputs: MERGED_MAP_OUTPUTS: 1 |
075 | ++++ GC time elapsed (ms): GC_TIME_MILLIS: 13 |
076 | ++++ CPU time spent (ms): CPU_MILLISECONDS: 3830 |
077 | ++++ Physical memory (bytes) snapshot: PHYSICAL_MEMORY_BYTES: 537718784 |
078 | ++++ Virtual memory (bytes) snapshot: VIRTUAL_MEMORY_BYTES: 7365263360 |
079 | ++++ Total committed heap usage (bytes): COMMITTED_HEAP_BYTES: 2022309888 |
080 | ========================================================== |
082 | * Counter Group: Shuffle Errors (Shuffle Errors) |
083 | number of counters in this group: 6 |
085 | ++++ BAD_ID: BAD_ID: 0 |
087 | ++++ CONNECTION: CONNECTION: 0 |
089 | ++++ IO_ERROR: IO_ERROR: 0 |
091 | ++++ WRONG_LENGTH: WRONG_LENGTH: 0 |
093 | ++++ WRONG_MAP: WRONG_MAP: 0 |
095 | ++++ WRONG_REDUCE: WRONG_REDUCE: 0 |
096 | ========================================================== |
098 | * Counter Group: File Input Format Counters (org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter) |
099 | number of counters in this group: 1 |
101 | ++++ Bytes Read: BYTES_READ: 81 |
102 | ========================================================== |
104 | * Counter Group: File Output Format Counters (org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter) |
105 | number of counters in this group: 1 |
106 | ++++ Bytes Written: BYTES_WRITTEN: 35 |
107 | ========================================================== |
109 | * Counter Group: tmp.WordCountWithCounter$WordsNature (tmp.WordCountWithCounter$WordsNature) |
110 | number of counters in this group: 3 |
112 | ++++ STARTS_WITH_DIGIT: STARTS_WITH_DIGIT: 1 |
113 | ++++ STARTS_WITH_LETTER: STARTS_WITH_LETTER: 3 |