大数据——mapreduce实际应用的案例（reduce全部聚合完成后，重新排序）

案例：
Log文件：

2017/07/28 qq.com/a
2017/07/28 qq.com/bx
2017/07/28 qq.com/by
2017/07/28 qq.com/by3
2017/07/28 qq.com/news
2017/07/28 sina.com/news/socail
2017/07/28 163.com/ac
2017/07/28 sina.com/news/socail
2017/07/28 163.com/sport
2017/07/28 163.com/ac
2017/07/28 sina.com/play
2017/07/28 163.com/sport
2017/07/28 163.com/ac
2017/07/28 sina.com/movie
2017/07/28 sina.com/play
2017/07/28 sina.com/movie
2017/07/28 163.com/sport
2017/07/28 sina.com/movie
2017/07/28 163.com/ac
2017/07/28 163.com/ac
2017/07/28 163.com/acc
2017/07/28 qq.com/by
2017/07/28 qq.com/by3
2017/07/28 qq.com/news
2017/07/28 163.com/sport
2017/07/28 sina.com/news/socail
2017/07/28 163.com/sport
2017/07/28 sina.com/movie
2017/07/28 sina.com/news/socail
2017/07/28 sina.com/movie
2017/07/28 qq.com/news
2017/07/28 163.com/bb
2017/07/28 163.com/cc
2017/07/28 sina.com/lady/
2017/07/28 163.com/cc
2017/07/28 qq.com/news
2017/07/28 qq.com/by
2017/07/28 qq.com/by3
2017/07/28 sina.com/lady/
2017/07/28 qq.com/by3
2017/07/28 sina.com/lady/
2017/07/28 qq.com/by3
2017/07/28 qq.com/news
2017/07/28 qq.com/by3
2017/07/28 163.com/sport
2017/07/28 163.com/sport
2017/07/28 sina.com/news/socail
2017/07/28 sina.com/lady/
2017/07/28 sina.com/play
2017/07/28 sina.com/movie
2017/07/28 sina.com/music
2017/07/28 sina.com/sport
2017/07/28 sina.com/sport
2017/07/28 163.com/sport
2017/07/28 sina.com/news/socail
2017/07/28 sohu.com/lady/
2017/07/28 sohu.com/play
2017/07/28 sohu.com/movie

需求：聚合所有网址被访问的次数，并得出访问量最大的前三个网址及其访问次数
分析：1、map阶段（key：网址名称，value：1）reduce阶段（key：网址名称，value：聚合后的总次数）
/2、如果需要对所有网址进行排序，就需要在全部的reduce完毕之后，重写reduce父类中的cleanup方法。
3、在reduce聚合的过程中，将结果的key和value封装到对象中，并将对象放到数组中，最后在cleanup方法中进行排序，和输出。

第一步：map逻辑

public class Logmapper extends Mapper<LongWritable,Text,Text,LongWritable> {
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String s = value.toString();
        String[] s1 = s.split(" ");
       String www = s1[1];
        context.write(new Text(www),new LongWritable(1));
    }
}

第二步：创建对象，并重写排序规则

public class User implements Comparable<User> {
    private int num;
    private String www;

    public User() {
    }

    public User(int num, String www) {
        this.num = num;
        this.www = www;
    }

    public int getNum() {
        return num;
    }

    public void setNum(int num) {
        this.num = num;
    }

    public String getWww() {
        return www;
    }

    public void setWww(String www) {
        this.www = www;
    }

    @Override
    public int compareTo(User o) {
        return  o.getNum() - this.num  ;
    }
}

第三步：reduce逻辑，并重写cleanup方法，进行结果对象排序。

public class Logreduceer extends Reducer<Text, LongWritable,Text,LongWritable> {



    List<User> list = new ArrayList<User>();



    @Override
    protected void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
        int a = 0;

        for (LongWritable value : values) {
            a += value.get();

        }
        User user = new User(a,key.toString());
        list.add(user);


    }




    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {

        Configuration cof = context.getConfiguration();
        int a = cof.getInt("top",3);

        Collections.sort(list);
        int num = 0;

        for (User user : list) {
            context.write(new Text(user.getWww()),new LongWritable(user.getNum()));
            num++;
            if(num==a){
                return;
            }

        }
    }
}

第四步：在main方法中，写job对象相关设置文件（main方法是执行操作的入口方法）

public class Submit01 {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

        Configuration configuration = new Configuration();



        Job job = Job.getInstance(configuration);

        //你要运行那个mian方法
        job.setJarByClass(Submit01.class);

        job.setMapperClass(Logmapper.class);
        job.setReducerClass(Logreduceer.class);

        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(LongWritable.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        File file = new File("f:\\hadoopdemo\\output2");
        if (file.exists()){
            FileUtil.fullyDelete(file);
        }

        FileInputFormat.addInputPath(job, new Path("F:\\test\\hadoopdemo\\request.dat"));
        FileOutputFormat.setOutputPath(job, new Path("f:\\hadoopdemo\\output2"));

        job.waitForCompletion(true);
        System.out.println(job.isSuccessful()?0:1);
    }
}

结果：

163.com/sport	8
baidu.com/music	7
sina.com/news/socail	7