复习
- hdfs的读写
- secondary namenode的工作原理
- shell脚本定时采集数据到hdfs
mapreduce
- 是一个编程框架
- 分为两个阶段:
- map阶段,task并发实例各司其职
- reduce阶段,task并发实例依然各司其职,但依赖第一阶段的task并发实例
- mapreduce只能分为1个map阶段和1个reduce阶段
- 可以通过多个mapreduce串联解决大型复杂问题
- mr application master的作用
实践
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
String[] words = line.split(" ");
for(String word:words){
context.write(new Text(word), new IntWritable(1));
}
}
}
2. reducer类
@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int count = 0;
for(IntWritable value:values){
count += value.get();
}
context.write(key, new IntWritable(count));
}
}
3. 定义一个主类,用来描述job并提交job
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job wcjob = Job.getInstance(conf);
wcjob.setJarByClass(WordCountRunner.class);
wcjob.setMapperClass(WordCountMapper.class);
wcjob.setReducerClass(WordCountReducer.class);
wcjob.setMapOutputKeyClass(Text.class);
wcjob.setMapOutputValueClass(IntWritable.class);
wcjob.setOutputKeyClass(Text.class);
wcjob.setOutputValueClass(IntWritable.class);
FileInputFormat.setInputPaths(wcjob, "hdfs://hdp-server01:9000/wordcount/data/big.txt");
FileOutputFormat.setOutputPath(wcjob, new Path("hdfs://hdp-server01:9000/wordcount/output/"));
boolean res = wcjob.waitForCompletion(true);
System.exit(res?0:1);
}
- 用debug追踪FileInputFormat()的执行