Hadoop partitioner及自定义partitioner

一、hadoop partitioner

所有partitioner都继承自抽象类Partitioner ，实现getPartition(KEY var1, VALUE var2, intvar3)，hadoop自带的partitioner有：

（1）TotalOrderPartitioner

一般做全局排序的时候使用

（2）KeyFieldBasedPartitioner

（3）BinaryPartitioner

public int getPartition(BinaryComparable key, V value, int numPartitions) {
    int length = key.getLength();
    int leftIndex = (this.leftOffset + length) % length;
    int rightIndex = (this.rightOffset + length) % length;
    int hash = WritableComparator.hashBytes(key.getBytes(), leftIndex, rightIndex - leftIndex + 1);
    return (hash & 2147483647) % numPartitions;
  }

（4）HashPartitioner 默认分区，通过ke的hash值确定分区
public int getPartition(K key, V value, int numReduceTasks) {
    return (key.hashCode() & 2147483647) % numReduceTasks;
  }


二、自定义paritioner
key为id，实现按key的末三位为分区
public class RTFPartitioner extends Partitioner<Text, Text> {
  @Override
  public int getPartition(Text key, Text value, int numPartitions) {
    String k = key.toString();
    int par = Integer.parseInt(k.substring(k.length() - 2));
    return par;
  }

}