1. 分区实践
我们学习MapReduce默认分区以及自定义分区实践
当我们要求将统计结果按照条件输出到不同文件(分区),比如按照统计结果将手机归属地不同省份输出到不同文件中(分区)
1.1 默认Partitioner分区
public class HashPartitioner<K, V> extends Partitioner<K, V> {
public int getPartition(K key, V value,
int numReduceTasks) {
return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
}
}
默认分区是根据key的hashCode对ReduceTasks[通过job.setNumReduceTasks(2)赋值]取模得到,用户没法控制key存储到哪个分区
1.2 自定义Partitioner分区
- 我们在resources目录下新建phone2.txt
1 13764368888 196.168.0.11 1116 854 200
2 13764368888 196.168.0.11 1136 834 200
3 13764368888 196.168.0.11 1146 824 200
4 13764368888 196.168.0.11 1116 804 200
5 13664368888 196.168.0.11 1116 854 200
6 13864368888 196.168.0.11 1136 834 200
7 13964368888 196.168.0.11 1146 824 200
8 13764368888 196.168.0.11 1116 804 200
- 新建自定义ProvincePartitioner类
public class ProvincePartitioner extends Partitioner<Text, FlowBean> {
@Override
public int getPartition(Text text, FlowBean flowBean, int numPartitions) {
// Text是手机号
String phone = text.toString().substring(0, 3);
// 注意分区号需要连续,从0开始分区
int partition;
if ("136".equals(phone)) {
partition = 0;
} else if ("137".equals(phone)) {
partition = 1;
} else if ("138".equals(phone)) {
partition = 2;
} else if ("139".equals(phone)) {
partition = 3;
} else {
partition = 4;
}
return partition;
}
}
- 新建FlowPartitionerDriver类
public class FlowPartitionerDriver {
public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "flowPartitioner");
job.setJarByClass(FlowPartitionerDriver.class);
job.setMapperClass(FlowMapper.class);
job.setCombinerClass(FlowReduce.class);
job.setReducerClass(FlowReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(FlowBean.class);
// 关联自定义分区类
job.setPartitionerClass(ProvincePartitioner.class);
// 设置ReduceTask任务数
job.setNumReduceTasks(5);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
- 传参运行
E:\Java\blogCode\hadoop\src\main\resources\phone2.txt E:\Java\blogCode\hadoop\src\main\resources\phone_ret2.txt
由图可知,产生了5个分区,实现了手机归属地不同省份输出到不同文件中
1.3 分区总结
- 如果ReduceTask数量>getPartition结果数,则会多产生空的part-r-000xx文件
- 如果1<ReduceTask数量<getPartition结果数,则有一部分数据无处写,会Exception
- 如果ReduceTask数量=1,则不管MapTask输出多少分区文件,最终结果都会交给一个ReduceTask,只会产生一个文件part-r-00000
- 分区号必须从零开始,逐一累加
2. 排序实践
本文我们学习MapReduce的全排序、二次排序以及区内排序
2.1 MapReduce概述
- MapTask和ReduceTask均会对数据按照key进行排序。该操作属于hadoop的默认行为。任何应用程序中的数据均会被排序,而不管逻辑上是否需要
- 默认排序是按照字典顺序排序,通过快速排序实现
- 对于MapTask,它会将处理结果暂时放到环形缓冲区中,当环形缓冲区使用率达到一定阈值后(默认80%),对缓冲区中的数据进行一次快速排序,并将这些有序数据溢写到磁盘上,而当数据处理完后,对磁盘上的所有文件进行归并排序
- 对于ReduceTask,会从每个MapTask上远程拷贝相应数据文件,如果文件大小超过阈值,则溢写磁盘上,否则存储到内存中;如果磁盘上文件数据达到阈值,则进行归并排序生成更大文件,如果内存中文件大小或数据超过阈值,则合并后溢写到磁盘
- 当所有数据拷贝完后,ReduceTask统一对内存和磁盘上的所有数据进行一次归并排序
2.2 全排序
最终输出结果只有一个,且文件内部有序。
// 我们主要实现WritableComparable接口重写compareTo方法即可
public class FlowBean implements WritableComparable<FlowBean> {
private long upFlow; // 上行流量
private long downFlow; // 下行流量
private long totalFlow; // 总流量
public FlowBean() {}
@Override
public void write(DataOutput out) throws IOException {
out.writeLong(upFlow);
out.writeLong(downFlow);
out.writeLong(totalFlow);
}
@Override
public void readFields(DataInput in) throws IOException {
this.upFlow = in.readLong();
this.downFlow = in.readLong();
this.totalFlow = in.readLong();
}
@Override
public String toString() {
return "FlowBean{" +
"upFlow=" + upFlow +
", downFlow=" + downFlow +
", totalFlow=" + totalFlow +
'}';
}
public long getUpFlow() {
return upFlow;
}
public void setUpFlow(long upFlow) {
this.upFlow = upFlow;
}
public long getDownFlow() {
return downFlow;
}
public void setDownFlow(long downFlow) {
this.downFlow = downFlow;
}
public long getTotalFlow() {
return totalFlow;
}
public void setTotalFlow() {
this.totalFlow = this.upFlow + this.downFlow;
}
@Override
public int compareTo(FlowBean o) {
// 按照总流量倒序排序
if (this.totalFlow > o.totalFlow) {
return -1;
}
if (this.totalFlow < o.totalFlow) {
return 1;
}
return 0;
}
}
// 注意我们将FlowBean作为键用于排序
public class FlowMapper extends Mapper<LongWritable, Text, FlowBean, Text> {
private FlowBean keyOut = new FlowBean();
private Text valueOut = new Text();
@Override
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, FlowBean, Text>.Context context) throws IOException, InterruptedException {
String line = value.toString();
String[] split = line.split(" ");
String phone = split[1];
String up = split[3];
String down = split[4];
valueOut.set(phone);
keyOut.setUpFlow(Long.parseLong(up));
keyOut.setDownFlow(Long.parseLong(down));
keyOut.setTotalFlow();
context.write(keyOut, valueOut);
}
}
public class FlowReduce extends Reducer<FlowBean, Text, Text, FlowBean> {
private FlowBean valueOut = new FlowBean();
@Override
protected void reduce(FlowBean key, Iterable<Text> values, Reducer<FlowBean, Text, Text, FlowBean>.Context context) throws IOException, InterruptedException {
for (Text value : values) {
context.write(value, key);
}
}
}
public class FlowDriver {
public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "flowSort");
job.setJarByClass(FlowDriver.class);
job.setMapperClass(FlowMapper.class);
job.setReducerClass(FlowReduce.class);
job.setOutputKeyClass(FlowBean.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
观察输出结果,可以看到已按结果降序输出
2.3 二次排序
在自定义排序过程中,如果compareTo中判断条件为两个即为二次排序
public int compareTo(FlowBean o) {
// 按照总流量倒序排序
if (this.totalFlow > o.totalFlow) {
return -1;
}
if (this.totalFlow < o.totalFlow) {
return 1;
}
// 按照下行流量倒序排序
if (this.downFlow > o.downFlow) {
return -1;
}
if (this.downFlow < o.downFlow) {
return 1;
}
return 0;
}
2.4 区内排序
MapReduce根据输入记录的键对数据集排序。保证输出的每个文件内部有序
// 注意FlowBean导入为排序的呢个类,最好新建包
public class ProvincePartitioner extends Partitioner<FlowBean, Text> {
@Override
public int getPartition(FlowBean flowBean, Text text, int numPartitions) {
// Text是手机号
String phone = text.toString().substring(0, 3);
// 注意分区号需要连续,从0开始分区
int partition;
if ("136".equals(phone)) {
partition = 0;
} else if ("137".equals(phone)) {
partition = 1;
} else if ("138".equals(phone)) {
partition = 2;
} else if ("139".equals(phone)) {
partition = 3;
} else {
partition = 4;
}
return partition;
}
}
public class FlowPartitionerDriver {
public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "flowPartitionerSort");
job.setJarByClass(FlowPartitionerDriver.class);
job.setMapperClass(FlowMapper.class);
job.setReducerClass(FlowReduce.class);
job.setMapOutputKeyClass(FlowBean.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(FlowBean.class);
// 关联自定义分区类
job.setPartitionerClass(ProvincePartitioner.class);
// 设置ReduceTask任务数,保持与分区数一致
job.setNumReduceTasks(5);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
3. Combiner实践
3.1 Combiner合并
- Combiner是MR程序中Mapper和Reducer之外的一种组件
- Combiner组件的父类就是Reducer
- Combiner和Reducer的区别在于运行的位置:Combiner在每一个MapTask所在的节点运行,Reducer是接收全局所有Mapper的输出结果
- Combiner意义在于对每个MapTask输出局部汇总,减少网络传输
- Combiner能够应用的前提是不影响最终业务逻辑,而且Combiner输出KV与Reducer输出KV类型需对应。如Combiner先求均值,然后传输到Reducer中再算最终均值结果是不一样的
其实我们在最初的WordCount案例中已经设置Combiner
public class WordCountDriver {
public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCountDriver.class);
job.setMapperClass(WordCountMapper.class);
// 复用WordCountReduce方法
job.setCombinerClass(WordCountReduce.class);
job.setReducerClass(WordCountReduce.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
3.2 NumReduceTasks为0情况
当我们设置job.setNumReduceTasks(0);
时,我们查看WordCount的输出结果为part-m-00000,只有Map阶段,没有Reduce阶段,也就没有所谓的Combiner合并
欢迎关注公众号算法小生与我沟通交流