mr实例分为两个阶段,一个是map阶段,一个是reduce阶段,中间用shuff来衔接,我们想运行mapreduce实例,只需要实现map业务和reduce业务逻辑即可。
map实现
//hadoop首先将input输入的文件内容split分为多份,每一份的内容用mapper.map来处理,其中Value就是需要处理的文本内容。context是上下文,用作连接和传递数据流的工具。
public class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {
//序列化的整数
private final IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
//将一段内容分解为一个个的字符串单词
StringTokenizer itr = new StringTokenizer(value.toString());
//将一个一个的单词和对应的数量反馈给context,context接收到之后,经过shuff之后,传递给reduce来整合。
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
reduce实现
public class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();
//key就是单词,也就是map中key,IntWritable就是一个个的value组成的列表。
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
//将单词出现的次数一个个的传递过来,然后相加得到结果。
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
提交job
public static void main(String[] args) throws Exception {
//使用默认配置
Configuration conf = new Configuration();
//解析命令行参数
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length < 2) {
System.out.println("Usage:wordcount");
System.exit(2);
}
//构造一个mr任务
Job job = Job.getInstance(conf, "word count");
//设置执行的jar
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
for (int i = 0; i < otherArgs.length - 1; ++i) {
FileInputFormat.addInputPath(job, new Path(otherArgs[i]));
}
FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
最后将代码打成jar包,用hadoop jar提交到mapreduce上,hadoop默认是mapreduce框架来执行,但我们可以修改默认配置,将其改为yarn。最终可以在yarn的监控界面展示任务执行的信息。