Mapreduce编程模型

mapreduce编程模型的分布式编程发的步骤
1)迭代 , 遍历输入数据 , 将数据解析成key/value对
2)将输入的key/value对,映射(map)成新的key/value对
3)根据key对中间数据进行分组(grouping)
4)以组为单位对数据进行规约(reduce)
5)迭代,将最终产生的key/value对输出到文件中
组件化和并行化InputFormat Mapper Partitioner Redece OutputForMat

.WordCount案例
.map
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
@Override
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context)
throws IOException, InterruptedException {
String[] split = value.toString().split("\s");
for (String word : split) {
context.write(new Text(word), new IntWritable(1));
}
}
}
reduce
public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
@Override
protected void reduce(Text key, Iterable values,
Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
int count = 0;
Iterator iterator = values.iterator();
while (iterator.hasNext()) {
IntWritable value = iterator.next();
count += value.get();
}
context.write(key, new IntWritable(count));
}
}

mapreduce运算模型默认在本地运行, 方便测试可以将mapreduce编写的业务代码在windows中测试

本地模式(默认模式)
默认运行在windows中 , 不需要任何的参数设置
public class WordCountJobSubmit {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
/conf.set(“mapreduce.framework.name”,“local”);
conf.set(“fs,defaultFS”, “file:///”);/
Job job = Job.getInstance(conf);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);

	job.setMapOutputKeyClass(Text.class);
	job.setMapOutputValueClass(IntWritable.class);
	
	job.setOutputKeyClass(Text.class);
	job.setOutputValueClass(IntWritable.class);
	
	FileInputFormat.setInputPaths(job, new Path("d:/data/wc/input"));
	FileOutputFormat.setOutputPath(job, new Path("d:/data/wc/output"));
	
	job.setNumReduceTasks(2);
	boolean b = job.waitForCompletion(true);
	System.exit(b?0:-1);
}

}

yarn模式
将MR程序运行在yarn中,保证集群运行正常 , yarn工作正常
public class JobSubmit {
public static void main(String[] args) throws Exception {
//设置操作的用户信息
System.setProperty(“HADOOP_USER_NAME” , “root”);
Configuration conf = new Configuration();
// 设置访问的集群的位置
conf.set(“fs.defaultFS”, “hdfs://doit01:9000”);
//设置yarn的位置
conf.set(“mapreduce.framework.name”, “yarn”);
//yarn的resourcemanager的位置
conf.set(“yarn.resourcemanager.hostname”, “doit01”);
//设置MapReduce程序运行在windows上的跨平台参数
conf.set(“mapreduce.app-submission.cross-platform”,“true”);
Job job = Job.getInstance(conf);
job.setJarByClass(JobSubmit.class);
//job.setJar(“C:\Users\ThinkPad\Desktop\wc.jar”);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);

	job.setMapOutputKeyClass(Text.class);
	job.setMapOutputValueClass(IntWritable.class);
	
	job.setOutputKeyClass(Text.class);
	job.setOutputValueClass(IntWritable.class);
	
	FileInputFormat.setInputPaths(job, new Path("/data/wc/input"));
	FileOutputFormat.setOutputPath(job, new Path("/data/wc/output"));
	
	job.setNumReduceTasks(2);
	boolean b = job.waitForCompletion(true);
	System.exit(b?0:-1);
}

}