mapreduce可以自定义的类

1. 分区partition

默认按照key的hashcode值分区。

自定义：手机号按照省份分区

• 自定义类，继承partitioner
• 重写getpartion方法

public class TextPartitioner extends Partitioner<Text, FlowBean> {

                  static HashMap<String, Integer> pm = new HashMap<String, Integer>();

                  static {

                                    pm.put("135", 0);

                                    pm.put("136", 1);

                                    pm.put("137", 2);

                                    pm.put("138", 3);

                                    pm.put("139", 4);

                  }

                  @Override

                  public int getPartition(Text key, FlowBean value, int numPartitions) {

                                    Integer code = pm.get(key.toString().substring(0, 3));

                                    return code == null ? 5 : code;

                  }

}

2. 分组group

默认按照key相同的分为一组

自定义：按照bean的id相同分为一组

• 自定义类，继承WritableComparator
• 重写compare方法

public class TextGroupComparator extends WritableComparator {

            @Override

            public int compare(WritableComparable a, WritableComparable b) {

                        OrderBean abean = (OrderBean) a;

                        OrderBean bbean = (OrderBean) b;

                        return abean.getId().compareTo(bbean.getId());

            }

}

3. 输入inputFormat

默认TextInputFormat

自定义：小文件合并

• 自定义类，继承FileInputFormat
• 改写内部类RecordReader

 

4. 输出outputFormat

默认TextOutputFormat

自定义：读取文件，根据url去匹配外部知识库。匹配到输出到一个目录下，匹配不到输出到另一个目录下

• 自定义类，继承FileOutputFormat
• 改写内部类RecordWriter，主要改写write方法