黑猴子的家：MapReduce流量汇总程序案例一

统计手机号耗费的总上行流量、下行流量、总流量（序列化）

1、数据

https://www.jianshu.com/p/bbebc7b959a8

输入数据格式

1363157993055   13560436666 C4-17-FE-BA-DE-D9:CMCC  120.196.100.99      18  15  1116        954     200
                手机号码                                                         上行流量     下行流量

输出数据格式

13560436666         1116              954           2070
手机号码             上行流量          下行流量        总流量

2、分析

基本思路

Map阶段
（1）读取一行数据，切分字段
（2）抽取手机号、上行流量、下行流量
（3）以手机号为key，bean对象为value输出，即context.write(手机号,bean);

Reduce阶段
（1）累加上行流量和下行流量得到总流量。
（2）实现自定义的bean来封装流量信息，并将bean作为map输出的key来传输
（3）MR程序在处理数据的过程中会对数据排序(map输出的kv对传输到reduce之前，会排序)，排序的依据是map输出的key
所以，我们如果要实现自己需要的排序规则，则可以考虑将排序因素放到key中，让key实现接口：WritableComparable。
然后重写key的compareTo方法。

3、编写mapreduce程序

（1）编写流量统计的bean对象

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.Writable;

// bean对象要实例化
public class FlowBean implements Writable {

    private long upFlow;
    private long downFlow;
    private long sumFlow;

    // 反序列化时，需要反射调用空参构造函数，所以必须有
    public FlowBean() {
        super();
    }

    public FlowBean(long upFlow, long downFlow) {
        super();
        this.upFlow = upFlow;
        this.downFlow = downFlow;
        this.sumFlow = upFlow + downFlow;
    }

    public long getSumFlow() {
        return sumFlow;
    }

    public void setSumFlow(long sumFlow) {
        this.sumFlow = sumFlow;
    }

    public long getUpFlow() {
        return upFlow;
    }

    public void setUpFlow(long upFlow) {
        this.upFlow = upFlow;
    }

    public long getDownFlow() {
        return downFlow;
    }

    public void setDownFlow(long downFlow) {
        this.downFlow = downFlow;
    }

    /**
     * 序列化方法
     * 
     * @param out
     * @throws IOException
     */
    @Override
    public void write(DataOutput out) throws IOException {
        out.writeLong(upFlow);
        out.writeLong(downFlow);
        out.writeLong(sumFlow);
    }

    /**
     * 反序列化方法 
     * 注意反序列化的顺序和序列化的顺序完全一致
     * @param in
     * @throws IOException
     */
    @Override
    public void readFields(DataInput in) throws IOException {
        upFlow = in.readLong();
        downFlow = in.readLong();
        sumFlow = in.readLong();
    }

    @Override
    public String toString() {
        return upFlow + "\t" + downFlow + "\t" + sumFlow;
    }
}

（2）编写mapreduce主程序

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class FlowCount {

   static class FlowCountMapper extends Mapper<LongWritable, Text, Text, FlowBean> {

       @Override
       protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
           // 1 将一行内容转成string
           String ling = value.toString();

           // 2 切分字段
           String[] fields = ling.split("\t");

           // 3 取出手机号码
           String phoneNum = fields[1];

           // 4 取出上行流量和下行流量
           long upFlow = Long.parseLong(fields[fields.length - 3]);
           long downFlow = Long.parseLong(fields[fields.length - 2]);

           // 5 写出数据
           context.write(new Text(phoneNum), new FlowBean(upFlow, downFlow));
       }
   }

   static class FlowCountReducer extends Reducer<Text, FlowBean, Text, FlowBean> {
       @Override
       protected void reduce(Text key, Iterable<FlowBean> values, Context context)
               throws IOException, InterruptedException {
           long sum_upFlow = 0;
           long sum_downFlow = 0;

           // 1 遍历所用bean，将其中的上行流量，下行流量分别累加
           for (FlowBean bean : values) {
               sum_upFlow += bean.getUpFlow();
               sum_downFlow += bean.getDownFlow();
           }

           // 2 封装对象
           FlowBean resultBean = new FlowBean(sum_upFlow, sum_downFlow);
           context.write(key, resultBean);
       }
   }

   public static void main(String[] args) throws Exception {
       // 1 获取配置信息，或者job对象实例
       Configuration configuration = new Configuration();
       Job job = Job.getInstance(configuration);

       // 6 指定本程序的jar包所在的本地路径
       job.setJarByClass(FlowCount.class);

       // 2 指定本业务job要使用的mapper/Reducer业务类
       job.setMapperClass(FlowCountMapper.class);
       job.setReducerClass(FlowCountReducer.class);

       // 3 指定mapper输出数据的kv类型
       job.setMapOutputKeyClass(Text.class);
       job.setMapOutputValueClass(FlowBean.class);

       // 4 指定最终输出的数据的kv类型
       job.setOutputKeyClass(Text.class);
       job.setOutputValueClass(FlowBean.class);

       // 5 指定job的输入原始文件所在目录
       FileInputFormat.setInputPaths(job, new Path(args[0]));
       FileOutputFormat.setOutputPath(job, new Path(args[1]));

       // 7 将job中配置的相关参数，以及job所用的java类所在的jar包， 提交给yarn去运行
       boolean result = job.waitForCompletion(true);
       System.exit(result ? 0 : 1);
   }
}

（3）将程序打成jar包，然后拷贝到hadoop集群中。
（4）启动hadoop集群
（5）执行flowcount程序

[victor@hadoop102 software]$ hadoop jar flowcount.jar com.victor.mr.flowsum.FlowCount /user/victor/flowcount/input/ /user/victor/flowcount/output

（6）查看结果

[victor@hadoop102 software]$ hadoop fs -cat /user/victor/flowcount/output/part-r-00000
13480253104 FlowBean [upFlow=180, downFlow=180, sumFlow=360]
13502468823 FlowBean [upFlow=7335, downFlow=110349, sumFlow=117684]
13560436666 FlowBean [upFlow=1116, downFlow=954, sumFlow=2070]
13560439658 FlowBean [upFlow=2034, downFlow=5892, sumFlow=7926]
13602846565 FlowBean [upFlow=1938, downFlow=2910, sumFlow=4848]

4、Code -> GitHub

https://github.com/liufengji/hadoop_mapreduce.git