mapreduce统计排序，页面访问次数

Mapper类：

public class PkMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        String[] split = line.split(",");
//      String [] split = StringUtil.split(line,' ');
        context.write(new Text(split[1].trim()), new IntWritable(1));
    }
}
复制代码

1. 拆分，注意key和value的值

2. 不遍历时key可以取数组的下标

Reducer类：

public class PkReducer extends Reducer<Text, IntWritable, Text, PageCount> {
    List<PageCount> sdf = new ArrayList();
    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int count = 0;
        for (IntWritable value : values) {
            count += value.get();
        }
        PageCount pageCount = new PageCount(key.toString(), count);
        sdf.add(pageCount);
    }

    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {

        Collections.sort(sdf);
        for (int i = 0; i < sdf.size(); i++) {
            context.write(new Text(sdf.get(i).getPage()), sdf.get(i));
            if (i == 3) {
                return;
            }
        }
    }
}
复制代码

1. 注意reduce输入的value值

2. 运用ArrayList数组存K，V，new封装类然后List（别名）.add添加

3. 重写MaperReduce的cleanup方法，运用Collections.sort进行排序

PageCount类：

public class PageCount implements Comparable<PageCount> {

    private String page;
    private int count;


    public PageCount() {
    }

    public PageCount(String page, int count) {
        this.page = page;
        this.count = count;
    }

    public String getPage() {
        return page;
    }

    public void setPage(String page) {
        this.page = page;
    }

    public int getCount() {
        return count;
    }

    public void setCount(int count) {
        this.count = count;
    }

    @Override
    public int compareTo(PageCount o) {
        return o.getCount() - this.count == 0 ? this.page.compareTo(o.getPage()) : o.getCount() - this.count;
    }

    @Override
    public String toString() {
        return String.valueOf(count);
    }
}
复制代码

1.注意无参构造函数的编写

2.注意运用Comparable<>的compareto方法进行正序和倒序

3.注意运用toString方法

Test类：

public class Tester {
    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        job.setJarByClass(Tester.class);

        job.setMapperClass(PkMapper.class);
        job.setReducerClass(PkReducer.class);

        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(IntWritable.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(PageCount.class);

        File file = new File("D:\\Documents\\Tencent Files\\1433214538\\FileRecv\\案例二页面访问次数\\练习用的数据\\shuchu");
        if (file.exists()){
            FileUtils.deleteDirectory(file);
        }

        FileInputFormat.setInputPaths(job,new Path("D:\\Documents\\Tencent Files\\1433214538\\FileRecv\\案例二页面访问次数\\练习用的数据\\input"));
        FileOutputFormat.setOutputPath(job,new Path("D:\\Documents\\Tencent Files\\1433214538\\FileRecv\\案例二页面访问次数\\练习用的数据\\shuchu"));

        job.setNumReduceTasks(1);

        job.waitForCompletion(true);
    }
}
复制代码