Hadoop中MultipleOutputs的应用,以及自定义Writable

版权声明:本文为博主原创文章,遵循 CC 4.0 by-sa 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/u011831754/article/details/53496548

MultipleOutputs的应用

①将结果生成至不同文件夹

private MultipleOutputs<Text, Text> outputs=null;
@Override
protected void setup(Context context) throws Exception {
    // TODO Auto-generated method stub
    outputs=new MultipleOutputs<Text, Text>(context);
}

String phoneNum=key.toString();
String baseOutputPath=phoneNum.substring(0, 3)+"/"+phoneNum.substring(3, 6)+"/";
Text value 
= new Text("上行:"+upTotal+"\t下行:"+downTotal+"\t总流量:"+(upTotal+downTotal));
outputs.write(key, value, baseOutputPath);

②通过不同形式生成结果

//命名说输出(JobSubmit)
MultipleOutputs.
addNamedOutput(job, "text", TextOutputFormat.class, Text.class, Text.class);
MultipleOutputs.
addNamedOutput(job, "seq", SequenceFileOutputFormat.class, Text.class, Text.class);
MultipleOutputs.
addNamedOutput(job, "map", MapFileOutputFormat.class, Text.class, Text.class);
//Reducer
outputs.write("text", key, value, baseOutputPath+"text/");
outputs.write("seq", key, value, baseOutputPath+"seq/");
outputs.write("map", key, value, baseOutputPath+"map/");

自定义Writable

//注意实现WritableComparable接口
public class FlowWritable implements WritableComparable<FlowWritable> {
//定义参数以及提供get、set方法
    private int upflow;
    private int downflow;
    public int getUpflow() {
        return upflow;
    }
    public void setUpflow(int upflow) {
        this.upflow = upflow;
    }
    public int getDownflow() {
        return downflow;
    }
    public void setDownflow(int downflow) {
        this.downflow = downflow;
    }
//提供write方法
    public void write(DataOutput out) throws IOException {
        // TODO Auto-generated method stub
        out.writeInt(upflow);
        out.writeInt(downflow);
    }
//提供readField方法
    public void readFields(DataInput in) throws IOException {
        // TODO Auto-generated method stub
        this.upflow=in.readInt();
        this.downflow=in.readInt();
    }
//实现compareTo方法
    public int compareTo(FlowWritable o) {
        // TODO Auto-generated method stub
        return 0;
    }
//覆盖toString方法
    @Override
    public String toString() {
        return  upflow + " " + downflow;
    }
}
展开阅读全文

没有更多推荐了,返回首页