1、shuffle阶段
shuffle,即洗牌的意思,在Map阶段到reduce阶段过程中,我们可以对数据进行分区、排序、规约、分组操作,这个过程会打乱其原有的顺序,具体如下
在MapTask到ReduceTask的过程,会经过网络,而这个过程会经过一次“洗牌”,也就是所谓的shuffle
更具体的MapReduce阶段可概括为下图
1.1分区(partition)
- 分区:将数据分成不同的文件,本质是将不同的键值对输出到不同的文件中
- 实现逻辑:在Map阶段对K2进行标记,根据一定的规则对k2进行分区
- 实现方式:将要分区的字段设置为k2,编写自定义类继承partition,根据实际情况设置ReduceTask个数
数据准备
flow.dat
复制以下数据,保存为
flow.dat
文件
文件解释:
文件以\t
进行字段分隔,从左到右的字段分别为:手机号、上行数据总包、下行数据总包、上行总流量、下行总流量
13480253104 3 3 180 180
13502468823 57 102 7335 110349
13560439658 33 24 2034 5892
13584138413 20 16 4116 1432
13600217502 37 266 2257 203704
13602846565 15 12 1938 2910
13660577991 24 9 6960 690
13719199419 4 0 240 0
13760778710 2 2 120 120
13823070001 6 3 360 180
13826544101 4 0 264 0
13922314466 12 12 3008 3720
13925057413 69 63 11058 48243
13926251106 4 0 240 0
13926435656 2 4 132 1512
15013685858 28 27 3659 3538
15920133257 20 20 3156 2936
15989002119 3 3 1938 180
18211575961 15 12 1527 2106
18320173382 21 18 9531 2412
根据手机号进行分区,要求135
开头的手机在一个文件,136
开头的手机在一个文件,137
开头的手机在一个文件,其余手机号码在一个文件
PartitionMapper.java
将k1,v1转为k2,v2的过程,其中k1是行偏移量,v1是这一行的数据,k2是手机号,在自定义分区中作为重要的依据,v2也是一行的数据
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
/**
* k1:行偏移量 v1:一行数据
* k2:手机号,v2:一整行数据
*/
public class PartitionMapper extends Mapper<LongWritable, Text, Text, Text> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String phone = value.toString().split("\t")[0];
context.write(new Text(phone), value);
}
}
PhonePartition.java
这里是分区的关键,分区是在map之后,所以接收到的数据自然是k2,v2
要注意的点是,返回的文件从0开始递增,按照这里的需求,文件自然是:0,1,2,3,而不能出现:0,1,2,4等的情况
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Partitioner;
/**
* partition在map后,所以取到的是k2v2
*/
public class PhonePartition extends Partitioner<Text, Text> {
/**
*
* @param k2 手机号码
* @param v2 一整段数据
* @param numPartitions 为reduce的个数
* @return return 0: part-r-00000 以此类推
*/
@Override
public int getPartition(Text k2, Text v2, int numPartitions) {
String phone = k2.toString();
if (phone.startsWith("135")) {
return 0;
} else if (phone.startsWith("136")) {
return 1;
} else if (phone.startsWith("137")) {
return 2;
} else {
return 3;
}
}
}
PartitionReducer.java
k2,v2转k3,v3的过程很简单,只需要将v2的值作为k3输出即可
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
/**
* k2:手机号 v2:一整行数据
* k3:一整行数据 v3:空
*/
public class PartitionReducer extends Reducer<Text, Text, Text, NullWritable> {
@Override
protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
for (Text value : values) {
context.write(value, NullWritable.get());
}
}
}
PartitionDriver
这里的代码后续可沿用,主要是
shuffle
注释中两段重要代码
job.setPartitionerClass(PhonePartition.class);
job.setNumReduceTasks(4);
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.net.URI;
public class PartitionDriver {
public static void main(String[] args) throws Exception{
Configuration configuration = new Configuration();
//1:创建一个Job对象
Job job = Job.getInstance(configuration, "FlowCount");
//2:对Job进行设置
//2.1 设置当前的主类的名字
job.setJarByClass(PartitionDriver.class);
//2.2 设置数据读取的路径
FileInputFormat.addInputPath(job,new Path("file:///D:\\bigdata\\input\\flowcount"));
//FileInputFormat.addInputPath(job,new Path(args[0]));
//2.3 指定你自定义的Mapper是哪个类及K2和V2的类型
job.setMapperClass(PartitionMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
//2.4 指定你自定义的Reducer是哪个类及K3和V3的类型
job.setReducerClass(PartitionReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);
//2.5 设置Shuffle阶段
job.setPartitionerClass(PhonePartition.class);
job.setNumReduceTasks(4);
//2.4 设置数据输出的路径--该目录要求不能存在,否则报错
Path outputPath = new Path("file:///D:\\bigdata\\output\\flowcount");
//Path outputPath = new Path(args[1]);
FileOutputFormat.setOutputPath(job,outputPath);
FileSystem fileSystem = FileSystem.get(new URI("file:///"), new Configuration());
boolean is_exists = fileSystem.exists(outputPath);
if(is_exists == true){
//如果目标文件存在,则删除
fileSystem.delete(outputPath,true);
}
//3:将Job提交为Yarn执行
boolean bl = job.waitForCompletion(true);
//4:退出任务进程,释放资源
System.exit(bl ? 0 : 1);
}
}
运行后,查看文件是否分为四个
part-r-00000
part-r-00001
part-r-00002
part-r-00003
1.2、排序
排序,一般是对某些字段进行升序或降序操作
MapReduce中只能根据K2进行排序
具体操作:将数据装进JavaBean,实现WritableComparable
接口中compareTo
方法
Flow.java
该类主要是对流量数据进行封装,再根据上行总流量进行倒序排序
主要是 实现compareTo
方法
注意:toString
方法要和文件中的分隔符一样,否则会按照toString
方法进行输出
import org.apache.hadoop.io.WritableComparable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
public class Flow implements WritableComparable<Flow> {
private Integer upFlow;
private Integer downFlow;
private Integer upCountFlow;
private Integer downCountFlow;
public Flow() {}
public Flow(Integer upFlow, Integer downFlow, Integer upCountFlow, Integer downCountFlow) {
this.upFlow = upFlow;
this.downFlow = downFlow;
this.upCountFlow = upCountFlow;
this.downCountFlow = downCountFlow;
}
@Override
public int compareTo(Flow o) {
return this.upCountFlow > o.upCountFlow ? -1 : 1;
}
@Override
public void write(DataOutput dataOutput) throws IOException {
dataOutput.writeInt(upFlow);
dataOutput.writeInt(downFlow);
dataOutput.writeInt(upCountFlow);
dataOutput.writeInt(downCountFlow);
}
@Override
public void readFields(DataInput dataInput) throws IOException {
upFlow = dataInput.readInt();
downFlow = dataInput.readInt();
upCountFlow = dataInput.readInt();
downCountFlow = dataInput.readInt();
}
public Integer getUpFlow() {
return upFlow;
}
public void setUpFlow(Integer upFlow) {
this.upFlow = upFlow;
}
public Integer getDownFlow() {
return downFlow;
}
public void setDownFlow(Integer downFlow) {
this.downFlow = downFlow;
}
public Integer getUpCountFlow() {
return upCountFlow;
}
public void setUpCountFlow(Integer upCountFlow) {
this.upCountFlow = upCountFlow;
}
public Integer getDownCountFlow() {
return downCountFlow;
}
public void setDownCountFlow(Integer downCountFlow) {
this.downCountFlow = downCountFlow;
}
@Override
public String toString() {
return upFlow + "\t" + downFlow + "\t" + upCountFlow + "\t" + downCountFlow;
}
}
SortMapper.java
由于排序是按照k2进行排序,所以k2必须是
Flow
类,v2由于字段的需要,是电话号码
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
/**
* k1:偏移量 v1:一行数据
* k2:flow(流量数据) v2:手机号码
*/
public class SortMapper extends Mapper<LongWritable, Text, Flow, Text> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] split = value.toString().split("\t");
String phone = split[0];
Flow flow = new Flow(Integer.parseInt(split[1]), Integer.parseInt(split[2]), Integer.parseInt(split[3]), Integer.parseInt(split[4]));
context.write(flow, new Text(phone));
}
}
SortReducer.java
Reduce阶段,将数据还原,即将k2变成v3,将v2变成k3
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
/**
* k2:flow(流量数据) v2:手机号码
* k3:手机号码 v3:flow(流量数据) ( 还原数据 )
*/
public class SortReduce extends Reducer<Flow, Text, Text, Flow> {
@Override
protected void reduce(Flow key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
for (Text value : values) {
context.write(value, key);
}
}
}
SortDriver.java
这里不想粘贴太多代码,将
1.1
中的PartitionDriver.java
中代码粘贴,剔除以下代码job.setPartitionerClass(PhonePartition.class); job.setNumReduceTasks(4);
其它就注意改Map和Reduce类以及对应的k、v类
1.3 规约
规约,本质上讲,是提前聚合。即在reduce之前,map之后,发生一次聚合,它是MapReduce中的优化手段,减少Map端和Reduce端网络传输的数据量。
拿wordcount案例来讲,其实可以在reduce之前提前做好统计,取到k2,v2的操作如下
规约的具体实现也很简单,只需要编写自定义类继承Reducer类,最后在Driver类中设置规约即可
// CombinerReducer即自定义类,具体写法与wordcount中的实现方式基本相同
job.setCombinerClass(CombinerReducer.class);
1.4、分组
分组的作用就是去K2进行去重,然后相同K2的V2存入集合;MR默认的分组是根据K2来决定的,相同K2的数据会被分到同一组;当默认的分组,不满足我们的需求时,我们可以使用自定义分组。
数据准备
student.dat
1,zhangsan,80
1,lisi,85
2,wangwu,79
1,zhaoliu,92
1,xiaoming,70
2,xiaohong,76
2,wanggang,88
2,xiaohei,90
1,ligang,95
要求:根据班级进行分组并获取每个班前三名的学生信息
Student.java
该类继承
WritableComparator
,实现了排序:按班级进行升序,如果班级相同,则通过成绩进行倒序排序
import org.apache.hadoop.io.WritableComparable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
public class Student implements WritableComparable<Student> {
private String clazz;
private String name;
private Integer grade;
public Student(String clazz, String name, Integer grade) {
this.clazz = clazz;
this.name = name;
this.grade = grade;
}
public Student() {}
public String getClazz() {
return clazz;
}
public void setClazz(String clazz) {
this.clazz = clazz;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public Integer getGrade() {
return grade;
}
public void setGrade(Integer grade) {
this.grade = grade;
}
@Override
public int compareTo(Student o) {
int result = this.clazz.compareTo(o.clazz);
// 班级相同,比较分数
if (result == 0) {
return (this.grade - o.grade) * -1;
}
return result;
}
@Override
public void write(DataOutput dataOutput) throws IOException {
dataOutput.writeUTF(clazz);
dataOutput.writeUTF(name);
dataOutput.writeInt(grade);
}
@Override
public void readFields(DataInput dataInput) throws IOException {
clazz = dataInput.readUTF();
name = dataInput.readUTF();
grade = dataInput.readInt();
}
}
GroupComparator.java
该类实现
WritableComparator
,重写compare
方法,根据班级进行分组
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableComparator;
public class GroupComparator extends WritableComparator {
public GroupComparator () {
//将k2类传给父类,并允许父类能够通过反射创建Student对象
super(Student.class, true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
Student a1 = (Student) a;
Student b1 = (Student) b;
return a1.getClazz().compareTo(b1.getClazz());
}
}
GroupMapper.java
主要以K2进行分组,对应分组的数据(v2)会聚合到一起,具体怎么取还看reduce操作
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
public class GroupMapper extends Mapper<LongWritable, Text, Student, Text> {
// 按k2进行分组
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] split = value.toString().split(",");
Student student = new Student(split[0], split[1], Integer.parseInt(split[2]));
context.write(student, value);
}
}
GroupReducer.java
相同的k2会进行聚合,所以取到的v2就是我们要的数据,具体取多少,前几用i进行递增操作
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
import java.io.IOException;
public class GroupReducer extends Reducer<Student, Text, Text, NullWritable> {
@Override
protected void reduce(Student key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
int i = 0;
for (Text value : values) {
context.write(value, NullWritable.get());
if (++i >= 3) {
break;
}
}
}
}
Driver
的编写
driver类的编写上面都大同小异,主要在shuffle注释下添加以下代码进行分组
job.setGroupingComparatorClass(GroupComparator.class);
至此,shuffle到此就告一段落了。