Hadoop_MapReduce_WritableComparable排序

目录

1.排序分类

2.自定义排序WritableComparable原理分析

3.WritableComparable排序案例实操（全排序）

1）需求

 2）需求分析

 3）代码实现

4.WritableComparable排序案例实操（二次排序）

1）代码

5.WritableComparable排序案例实操（区内排序）

1）需求

2）需求分析

3）代码 

---

1.排序分类

2.自定义排序WritableComparable原理分析

bean对象做为key传输，需要实现WritableComparable接口重写compareTo方法，就可以实现排序。

@Override
public int compareTo(FlowBean bean) {

	int result;
		
	// 按照总流量大小，倒序排列
	if (this.sumFlow > bean.getSumFlow()) {
		result = -1;
	}else if (this.sumFlow < bean.getSumFlow()) {
		result = 1;
	}else {
		result = 0;
	}

	return result;
}

3.WritableComparable排序案例实操（全排序）

1）需求

根据之前序列化案例产生的结果再次对总流量进行倒序排序。

（1）输入数据：之前序列化案例产生的结果，注意输出文件夹要删除除了part-r-00000的文件！

（2）期望输出数据

 2）需求分析

 3）代码实现

（1）FlowBean对象在在需求1基础上增加了比较功能

/**
 * 1、定义类实现writable接口
 * 2、重写序列化和反序列化方法
 * 3、重写空参构造
 * 4、toString方法
 */
public class FlowBean implements WritableComparable<FlowBean> {
    private long upFlow;
    private long downFlow;
    private long sumFlow;

    public FlowBean() {
    }

    public long getUpFlow() {
        return upFlow;
    }

    public void setUpFlow(long upFlow) {
        this.upFlow = upFlow;
    }

    public long getDownFlow() {
        return downFlow;
    }

    public void setDownFlow(long downFlow) {
        this.downFlow = downFlow;
    }

    public long getSumFlow() {
        return sumFlow;
    }

    public void setSumFlow(long sumFlow) {
        this.sumFlow = sumFlow;
    }

    public void setSumFlow() {
        this.sumFlow = this.upFlow + this.downFlow;
    }

    @Override
    public void write(DataOutput out) throws IOException {
        out.writeLong(upFlow);
        out.writeLong(downFlow);
        out.writeLong(sumFlow);
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        //序列化和反序列化顺序一致
        this.upFlow = in.readLong();
        this.downFlow = in.readLong();
        this.sumFlow = in.readLong();
    }

    @Override
    public String toString() {
        return upFlow + "\t" + downFlow + "\t" + sumFlow;
    }

    @Override
    public int compareTo(FlowBean o) {
        //按总流量的倒序排序
        if(this.sumFlow > o.sumFlow){
            return -1;//倒序返回-1，正序返回1
        }else if(this.sumFlow < o.sumFlow){
            return 1;
        }else{
            return 0;
        }
    }
}

（2）编写Mapper类

Map输出变成<FlowBean,Text>，因为要比较的是FlowBean流量。

//FlowBean作为key，Text作为value
public class FlowMapper extends Mapper<LongWritable, Text, FlowBean, Text> {
    private FlowBean outK = new FlowBean();
    private Text outV = new Text();

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        //获取一行
        String line = value.toString();

        //切割
        String[] split = line.split("\t");

        //封装
        outV.set(split[0]);
        outK.setUpFlow(Long.parseLong(split[1]));
        outK.setDownFlow(Long.parseLong(split[2]));
        outK.setSumFlow();

        //写出
        context.write(outK,outV);
    }
}

（3）编写Reducer类

//最终想输出的是手机号 流量，因此输出key为Text，输出value为FlowBean
public class FlowReducer extends Reducer<FlowBean, Text, Text, FlowBean> {
    @Override
    protected void reduce(FlowBean key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
        //key相同，不同的value，即总流量相同，不同的手机号
        //不同key时调用一次reduce方法
        for (Text value : values) {
            //调换KV位置,反向写出
            context.write(value,key);
        }
    }
}

（4）编写Driver类

public class FlowDriver {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

        //1 获取job对象
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        //2 关联本Driver类
        job.setJarByClass(FlowDriver.class);

        //3 关联Mapper和Reducer
        job.setMapperClass(FlowMapper.class);
        job.setReducerClass(FlowReducer.class);

        //4 设置Map端输出数据的KV类型
        job.setMapOutputKeyClass(FlowBean.class);
        job.setMapOutputValueClass(Text.class);

        //5 设置程序最终输出的KV类型
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(FlowBean.class);

        //在之前案例的输出进行排序，需要删除文件夹下的除part-r-00000外的三个文件
        FileInputFormat.setInputPaths(job,new Path("D:\\code\\Hadoop\\output4"));
        FileOutputFormat.setOutputPath(job,new Path("D:\\code\\Hadoop\\output6"));

        boolean result = job.waitForCompletion(true);
        System.exit(result? 0:1);
    }

}

4.WritableComparable排序案例实操（二次排序）

1）代码

基本和全排序一样，只有FlowBean对象重写WritableComparable接口的方法时多了一个判断

    @Override
    public int compareTo(FlowBean o) {
        //按总流量的倒序排序
        if(this.sumFlow > o.sumFlow){
            return -1;//倒序返回-1，正序返回1
        }else if(this.sumFlow < o.sumFlow){
            return 1;
        }else{
            //二次排序
            //总流量相等的情况下按上流量正序排
            if(this.upFlow > o.upFlow){
                return 1;
            }else if(this.upFlow < o.upFlow){
                return -1;
            }else{
                return 0;
            }
        }
    }

5.WritableComparable排序案例实操（区内排序）

1）需求

要求每个省份手机号输出的文件中按照总流量内部排序。

2）需求分析

在二次排序代码基础上增加自定义分区类，分区按照省份手机号设置。排序加上分区。

3）代码 

（1）增加自定义分区类

public class ProvincePartitioner2 extends Partitioner<FlowBean, Text> {
    @Override
    public int getPartition(FlowBean flowBean, Text text, int i) {
        String phone = text.toString();

        String prePhone = phone.substring(0, 3);

        int partition;
        if("136".equals(prePhone)){
            partition = 0;
        }else if("137".equals(prePhone)){
            partition = 1;
        }else if("138".equals(prePhone)){
            partition = 2;
        }else if("139".equals(prePhone)){
            partition = 3;
        }else{
            partition = 4;
        }
        return partition;
    }
}

（2）在驱动类中添加分区类

public class FlowDriver {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

        //1 获取job对象
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        //2 关联本Driver类
        job.setJarByClass(FlowDriver.class);

        //3 关联Mapper和Reducer
        job.setMapperClass(FlowMapper.class);
        job.setReducerClass(FlowReducer.class);

        //4 设置Map端输出数据的KV类型
        job.setMapOutputKeyClass(FlowBean.class);
        job.setMapOutputValueClass(Text.class);

        //5 设置程序最终输出的KV类型
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(FlowBean.class);

        job.setPartitionerClass(ProvincePartitioner2.class);
        job.setNumReduceTasks(5);

        //在之前案例的输出进行排序，需要删除文件夹下的除part-r-00000外的三个文件
        FileInputFormat.setInputPaths(job,new Path("D:\\code\\Hadoop\\output4"));
        FileOutputFormat.setOutputPath(job,new Path("D:\\code\\Hadoop\\output7"));

        boolean result = job.waitForCompletion(true);
        System.exit(result? 0:1);
    }
}