一、hadoop partitioner
所有partitioner都继承自抽象类Partitioner ,实现getPartition(KEY var1, VALUE var2, intvar3),hadoop自带的partitioner有:
(1)TotalOrderPartitioner
一般做全局排序的时候使用
(2)KeyFieldBasedPartitioner
(3)BinaryPartitioner
public int getPartition(BinaryComparable key, V value, int numPartitions) { int length = key.getLength(); int leftIndex = (this.leftOffset + length) % length; int rightIndex = (this.rightOffset + length) % length; int hash = WritableComparator.hashBytes(key.getBytes(), leftIndex, rightIndex - leftIndex + 1); return (hash & 2147483647) % numPartitions; }
(4)HashPartitioner 默认分区,通过ke的hash值确定分区
public int getPartition(K key, V value, int numReduceTasks) { return (key.hashCode() & 2147483647) % numReduceTasks; }
二、自定义paritioner
key为id,实现按key的末三位为分区
public class RTFPartitioner extends Partitioner<Text, Text> { @Override public int getPartition(Text key, Text value, int numPartitions) { String k = key.toString(); int par = Integer.parseInt(k.substring(k.length() - 2)); return par; } }