MapReduce的全局计数器
1.1、介绍
计数器是用来记录job的执行进度和状态的。它的作用可以理解为日志。我们可以在程序的某个位置插入计数器,记录数据或者进度的变化情况。
MapReduce 计数器(Counter)为我们提供一个窗口,用于观察 MapReduce Job 运行期的各种细节数据。对MapReduce性能调优很有帮助,MapReduce性能优化的评估大部分都是基于这些 Counter 的数值表现出来的。
MapReduce 自带了许多默认Counter,现在我们来分析这些默认Counter 的含义,方便大家观察 Job 结果,如输入的字节数、输出的字节数、Map端输入/输出的字节数和条数、Reduce端的输入/输出的字节数和条数等
1.2、需求
在实际生产代码中,常常需要将数据处理过程中遇到的不合规数据行进行全局计数,类似这种需求可以借助MapReduce框架中提供的全局计数器来实现
1.3、场景介绍
现在我们通过全局计数器来实现,统计一个目录下的所有文件或者一个文件中的总行数和总单词数
1.4、代码实现
package com.ghgj.mazh.mapreduce.counter;
import java.io.IOException;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Counter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class WordAndLineCounter {
/**
* mapreduce全局计数器,是用枚举类进行计数器的定义
* COUNT_LINES用来统计总行数,
* COUNT_WORDS用来统计总单词数
*/
enum MyCouterWordCount {
COUNT_WORDS, COUNT_LINES
}
// 驱动程序
public static void main(String[] args) throws Exception {
// 指定hdfs相关的参数
Configuration conf = new Configuration();
Job job = Job.getInstance(conf);
// 设置jar包所在路径
job.setJarByClass(WordAndLineCounter.class);
job.setMapperClass(WordAndLineCounterMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
// 本地路径
Path inputPath = new Path("D:\\bigdata\\wordcount\\input");
FileInputFormat.setInputPaths(job, inputPath);
Path outputPath = new Path("D:\\bigdata\\wordcount\\output");
FileSystem fs = FileSystem.get(conf);
if (fs.exists(outputPath)) {
fs.delete(outputPath, true);
}
FileOutputFormat.setOutputPath(job, outputPath);
// 最后提交任务
boolean waitForCompletion = job.waitForCompletion(true);
System.exit(waitForCompletion ? 0 : 1);
}
private static class WordAndLineCounterMapper extends Mapper<LongWritable, Text, Text, LongWritable> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
// 获取计数器,行数计数器
Counter counter = context.getCounter(MyCouterWordCount.COUNT_LINES);
// 计数器计数
counter.increment(1L);
String[] words = value.toString().split(" ");
for (String word : words) {
if(!StringUtils.isBlank(word)){
context.getCounter(MyCouterWordCount.COUNT_WORDS).increment(1L);
}
}
}
}
}
1.5、输入数据
hello huangbo
hello xuzheng
hello wangbaoqiang
one two three four five
one two three four
one two three
one two
hello hi
1.6、计算结果
INFO [main] - Job job_local1030848363_0001 completed successfully
INFO [main] - Counters: 35
File System Counters
FILE: Number of bytes read=614
FILE: Number of bytes written=506970
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=8
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=104
Combine input records=0
Combine output records=0
Reduce input groups=0
Reduce shuffle bytes=6
Reduce input records=0
Reduce output records=0
Spilled Records=0
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=514850816
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
com.ghgj.mazh.mapreduce.counter.WordAndLineCounter$MyCouterWordCount
COUNT_LINES=8
COUNT_WORDS=22
File Input Format Counters
Bytes Read=127
File Output Format Counters
Bytes Written=8
这就是最后的计算结果:在上面结果中的倒数第5-7行
com.ghgj.mazh.mapreduce.counter.WordAndLineCounter$MyCouterWordCount
COUNT_LINES=8
COUNT_WORDS=22
各位看官,有没有看懂的,如果有疑问的地方,可以举手提问。