细节决定成败 MapReduce任务实战 Reduce Join

一、任务描述

在hdfs 上有两个文件

学生信息文件：hdfs://***.***.***:8020/user/train/joinjob/student.txt

以逗号分隔，第一列为学号，学号每个学生唯一，第二列为学生姓名。

点击(此处)折叠或打开

1. 2016001,Join
   
2. 2016002,Abigail
   
3. 2016003,Abby
   
4. 2016004,Alexandra
   
5. 2016005,Cathy
   
6. 2016006,Katherine

学生成绩信息文件：hdfs://***.***.***:8020/user/train/joinjob/student_score.txt

以逗号分隔，第一列为学号，学号每个学生唯一，第二列为课程代码，第三列为分数。

点击(此处)折叠或打开

1. 2016001,YY,60
   
2. 2016001,SX,88
   
3. 2016001,YW,91
   
4. 2016002,SX,77
   
5. 2016002,YW,33
   
6. .............

期望的结果文件形式如下

点击(此处)折叠或打开

1. 2016001,Join,YY,60
   
2. 2016001,Join,SX,88
   
3. 2016001,Join,YW,91
   
4. 2016002,Abigail,SX,77
   
5. 2016002,Abigail,YW,33
   
6. 2016002,Abigail,YY,56
   
7. 2016003,Abby,YY,34
   
8. 2016003,Abby,SX,84
   
9. 2016003,Abby,YW,69
   
10. 2016004,Alexandra,YY,89
    
11. 2016004,Alexandra,SX,84
    
12. .......

二、任务分析

1. 这是一个两个数据集关联的任务。关联有map端join ，和reduce端join。 通常在有一个数据集比较小，可在全部放到内存中时，选择map端join；
2. 当两个数据集都很大,不能把一个数据集合部放到内存中时，使用reduce端join，这时也可以使用Bloom过滤器来增加效率。
3. 本次任务我们使用reduce端join实现。
4. 涉及到的内容有：自定义KEY，自定义GroupingComparator,配置多个Mapper类，配置多个输入路径

5. 三、实现思路

   1. 分别编写两个Mapper：
   2. StudentMapper 用来读取和处理student.txt中的内容。
   3. ScoreMapper 用来读取和处理student_scor.txt中的内容。
   4. 这两个mapper的输入都要进行到JoinReducer的reduce函数中处理。
   5. 两个mapper中输出的相同的学生信息进入到同一次reduce调用中，在其中进行关联，同时又要能标记每一条记录是来自学生信息，还是成绩信息。
   6. 还有有可能同一个学号的数据量有很大，不能在reduce中先存下来再处理，所以把学生信息的value放到前面，成绩信息放到后面。
   7. 就可以在一开始遍历values时就可以取到学生姓名，然后把姓名保存起来，后面遍历成绩信息时直接使用姓名值。
   8. 综上分析以如下方法完成此任务：
   9. 1、定义一个自定义KEY,StuentCustomKey, 属性有学生ID，和信息来源（标记此记录来源于学生信息表，还是成绩表），实现comparaTo方法，确保按学号排升序，按source排降序 （student要排在score前面）
   10. 2、StudentMapper 用来读取和处理student.txt中的内容，输出StuentCustomKey、Text。其中设置StuentCustomKey的source值为“student”
   11. 3、ScoreMapper 用来读取和处理student_scor.txt中的内容,输出StuentCustomKey、Text。其中设置StuentCustomKey的source值为“score”
   12. 4、为了保证相同学号的信息进入同一次reduce调用，要实现一个自定义GroupingComparator，实现中使用sid进行比较。（注意自定义GroupingComparator的构造）

   13. 三、实现代码

       
       
   14. 自定义KEY
       

       点击(此处)折叠或打开

       1. package join;
          
       2. 
          
       3. import java.io.DataInput;
          
       4. import java.io.DataOutput;
          
       5. import java.io.IOException;
          
       6. 
          
       7. import org.apache.hadoop.io.WritableComparable;
          
       8. 
          
       9. public class StudentCustomKey implements WritableComparable<StudentCustomKey>{
          
       10.         private String sid;
           
       11.         private String source;
           
       12. 
           
       13.         @Override
           
       14.         public void write(DataOutput out) throws IOException {
           
       15.                 out.writeUTF(sid);
           
       16.                 out.writeUTF(source);
           
       17.         }
           
       18. 
           
       19.         @Override
           
       20.         public void readFields(DataInput in) throws IOException {
           
       21.                 this.sid = in.readUTF();
           
       22.                 this.source = in.readUTF();
           
       23.         }
           
       24. 
           
       25.         @Override
           
       26.         public int compareTo(StudentCustomKey key) {
           
       27.                 int r= this.sid.compareTo(key.sid);
           
       28.                 if(r==0)
           
       29.                         r= -this.source.compareTo(key.source);
           
       30.                 return r;
           
       31.         }
           
       32. 
           
       33.         public String getSid() {
           
       34.                 return sid;
           
       35.         }
           
       36. 
           
       37.         public void setSid(String sid) {
           
       38.                 this.sid = sid;
           
       39.         }
           
       40. 
           
       41.         public String getSource() {
           
       42.                 return source;
           
       43.         }
           
       44. 
           
       45.         public void setSource(String source) {
           
       46.                 this.source = source;
           
       47.         }
           
       48. 
           
       49.         @Override
           
       50.         public String toString(){
           
       51.                 return this.sid+","+this.source;
           
       52.         }
           
       53. 
           
       54. }

       
       自定义 GroupingComparator
       
       

       点击(此处)折叠或打开

       1. package join;
          
       2. 
          
       3. import org.apache.hadoop.io.WritableComparable;
          
       4. import org.apache.hadoop.io.WritableComparator;
          
       5. 
          
       6. public class GroupComparator extends WritableComparator {
          
       7. 
          
       8.         public GroupComparator(){
          
       9.                 super(StudentCustomKey.class,true); //mark
          
       10.         }
           
       11. 
           
       12.         @Override
           
       13.         public int compare(WritableComparable a, WritableComparable b) {
           
       14.                 StudentCustomKey key1 = (StudentCustomKey)a;
           
       15.                 StudentCustomKey key2 = (StudentCustomKey)b;
           
       16.                 return key1.getSid().compareTo(key2.getSid());
           
       17.         }
           
       18. 
           
       19. }

       
       Job类，包括两个Mapper 和一个Reducer。
       
       

       点击(此处)折叠或打开

       1. package join;
          
       2. 
          
       3. import java.io.IOException;
          
       4. 
          
       5. import org.apache.commons.lang.StringUtils;
          
       6. import org.apache.hadoop.conf.Configuration;
          
       7. import org.apache.hadoop.conf.Configured;
          
       8. import org.apache.hadoop.fs.FileSystem;
          
       9. import org.apache.hadoop.fs.Path;
          
       10. import org.apache.hadoop.io.LongWritable;
           
       11. import org.apache.hadoop.io.NullWritable;
           
       12. import org.apache.hadoop.io.Text;
           
       13. import org.apache.hadoop.mapreduce.Job;
           
       14. import org.apache.hadoop.mapreduce.Mapper;
           
       15. import org.apache.hadoop.mapreduce.Mapper.Context;
           
       16. import org.apache.hadoop.mapreduce.Reducer;
           
       17. import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;
           
       18. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
           
       19. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
           
       20. import org.apache.hadoop.util.Tool;
           
       21. import org.apache.hadoop.util.ToolRunner;
           
       22. import org.apache.log4j.Logger;
           
       23. 
           
       24. public class ReduceJoinJob extends Configured implements Tool {
           
       25. 
           
       26.         public static class StudentMapper extends Mapper<LongWritable,Text,StudentCustomKey,Text>{
           
       27.                 private StudentCustomKey newKey = new StudentCustomKey();
           
       28.                 @Override
           
       29.                 protected void map(LongWritable key, Text value, Context context)
           
       30.                                 throws IOException, InterruptedException {
           
       31.                         String [] words = StringUtils.split(value.toString(),',');
           
       32.                         newKey.setSid(words[0]);
           
       33.                         newKey.setSource("student");
           
       34.                         context.write(newKey, value);
           
       35.                 }
           
       36. 
           
       37.         }
           
       38. 
           
       39.         public static class ScoreMapper extends Mapper<LongWritable,Text,StudentCustomKey,Text>{
           
       40.                 private StudentCustomKey newKey = new StudentCustomKey();
           
       41.                 @Override
           
       42.                 protected void map(LongWritable key, Text value, Context context)
           
       43.                                 throws IOException, InterruptedException {
           
       44.                         String [] words = StringUtils.split(value.toString(),',');
           
       45.                         newKey.setSid(words[0]);
           
       46.                         newKey.setSource("score");
           
       47.                         context.write(newKey, value);
           
       48.                 }
           
       49.         }
           
       50. 
           
       51.         public static class JoinReducer extends Reducer<StudentCustomKey,Text,NullWritable,Text>{
           
       52.                 Logger log = Logger.getLogger(getClass());
           
       53.                 private Text newValue = new Text();
           
       54.                 @Override
           
       55.                 protected void reduce(StudentCustomKey key, Iterable<Text> values,Context context)
           
       56.                                 throws IOException, InterruptedException {
           
       57.                         String name = "";
           
       58.                         for(Text v : values){
           
       59.                                 if(key.getSource().equals("student")){
           
       60.                                         name = StringUtils.split(v.toString(),',')[1];
           
       61.                                         continue;
           
       62.                                 }
           
       63.                                 newValue.set(name +","+ v.toString());
           
       64.                                 context.write(NullWritable.get(), newValue);
           
       65.                         }
           
       66. 
           
       67.                 }
           
       68. 
           
       69.         }
           
       70.         @Override
           
       71.         public int run(String[] args) throws Exception {
           
       72.                 Job job = Job.getInstance(getConf(),"ReduceJoinJob");
           
       73.                 job.setJarByClass(getClass());
           
       74.                 Configuration conf = job.getConfiguration();
           
       75. 
           
       76.                 MultipleInputs.addInputPath(job, new Path("joinjob/student.txt"), TextInputFormat.class, StudentMapper.class);
           
       77.                 MultipleInputs.addInputPath(job, new Path("joinjob/student_score.txt"), TextInputFormat.class, ScoreMapper.class);
           
       78.                 Path out = new Path("joinjob/output1");
           
       79.                 TextOutputFormat.setOutputPath(job, out);
           
       80.                 FileSystem.get(conf).delete(out, true);
           
       81. 
           
       82.                 job.setOutputFormatClass(TextOutputFormat.class);
           
       83.                 job.setReducerClass(JoinReducer.class);
           
       84. 
           
       85.                 job.setMapOutputKeyClass(StudentCustomKey.class);
           
       86.                 job.setMapOutputValueClass(Text.class);
           
       87.                 job.setOutputKeyClass(NullWritable.class);
           
       88.                 job.setOutputValueClass(Text.class);
           
       89. 
           
       90.                 job.setGroupingComparatorClass(GroupComparator.class);
           
       91.                 return job.waitForCompletion(true)?0:1;
           
       92.         }
           
       93.         public static void main(String [] args){
           
       94.                 int r=0;
           
       95.                 try{
           
       96.                         r= ToolRunner.run(new Configuration(), new ReduceJoinJob() , args);
           
       97.                 }catch(Exception e){
           
       98.                         e.printStackTrace();
           
       99.                 }
           
       100.                 System.exit(r);
            
       101.         }
            
       102. }

       三、打包运行

       
       

   15. 结果如下。
       

       点击(此处)折叠或打开

       1. Join,2016001,YY,60
          
       2. Join,2016001,SX,88
          
       3. Join,2016001,YW,91
          
       4. Abigail,2016002,SX,77
          
       5. Abigail,2016002,YW,33
          
       6. Abigail,2016002,YY,56
          
       7. Abby,2016003,YW,69
          
       8. Abby,2016003,SX,84
          
       9. Abby,2016003,YY,34
          
       10. Alexandra,2016004,YW,100
           
       11. Alexandra,2016004,SX,84
           
       12. Alexandra,2016004,YY,89
           
       13. Cathy,2016005,YW,63
           
       14. Cathy,2016005,YY,53
           
       15. Cathy,2016005,SX,43
           
       16. Katherine,2016006,SX,90
           
       17. Katherine,2016006,YW,90
           
       18. Katherine,2016006,YY,90
       相关博文： Hadoop的GroupComparator是如何起做用的（源码分析）
   16. 本文地址：http://blog.itpub.net/30066956/viewspace-2120133/
   
   
6. 
   

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/30066956/viewspace-2120133/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/30066956/viewspace-2120133/