今天做了个用MapReduce对数据进行二次排序,这里的重点在于排序,所以要重写WritableComparator的排序方法compare方法,这里要注意一下的返回值如果返回-1就是升序排序,返回1就是降序排序。谨以此文章来记录自己的学习之路及希望能帮助到有需要的朋友。
题目要求:
|1. 综合设计题
假设有两个文件代表两个班级的成绩,扩展名为.csv,文件中有两列数据,代表学号及数学成绩。具体文件数据如下。
说明:班级信息请参考学号,如2017876211,前八位为班级区分信息,后两位为学号信息。
程序设计要求如下:
要求输出第一列为学号,第二列为数学成绩。
要求输出按数学成绩降序排序,如果数学成绩相同,按学号升序排序(二次排序)。
要求每个班级输出一个文件。
数据如下:
第一份数据:
2017876101,39
2017876102,100
2017876103,49
2017876104,79
2017876105,84
2017876106,93
2017876107,24
2017876108,99
2017876109,52
2017876110,43
2017876111,13
2017876112,74
2017876113,88
2017876114,48
2017876115,58
2017876116,93
2017876117,43
2017876118,51
2017876119,91
2017876120,93
2017876121,21
2017876122,47
2017876123,16
2017876124,19
2017876125,93
2017876126,93
2017876127,20
2017876128,16
2017876129,21
2017876130,23
2017876131,87
2017876132,79
2017876133,49
2017876134,72
2017876135,93
2017876136,79
2017876137,87
2017876138,21
2017876139,15
2017876140,63
2017876141,28
2017876142,19
2017876143,86
2017876144,48
2017876145,65
2017876146,98
2017876147,88
2017876148,72
2017876149,14
2017876150,26
2017876151,72
2017876152,87
2017876153,99
2017876154,99
2017876155,100
2017876156,100
2017876157,35
2017876158,35
2017876159,72
2017876160,72
2017876161,63
2017876162,46
2017876163,55
2017876164,41
2017876165,84
2017876166,27
2017876167,51
2017876168,44
2017876169,82
2017876170,32
2017876171,36
2017876172,72
2017876173,92
2017876174,95
2017876175,87
2017876176,95
2017876177,27
2017876178,91
2017876179,56
2017876180,59| |
第二份数据:
2017876201,96
2017876202,84
2017876203,96
2017876204,26
2017876205,67
2017876206,96
2017876207,96
2017876208,55
2017876209,19
2017876210,54
2017876211,47
2017876212,96
2017876213,69
2017876214,90
2017876215,38
2017876216,31
2017876217,72
2017876218,46
2017876219,37
2017876220,14
2017876221,96
2017876222,62
2017876223,96
2017876224,77
2017876225,40
2017876226,48
2017876227,20
2017876228,32
2017876229,89
2017876230,98
2017876231,88
2017876232,64
2017876233,88
2017876234,80
2017876235,96
2017876236,88
2017876237,64
2017876238,100
2017876239,15
2017876240,93
2017876241,44
2017876242,48
2017876243,72
2017876244,83
2017876245,62
2017876246,60
2017876247,81
2017876248,44
2017876249,100
2017876250,88
2017876251,22
2017876252,22
2017876253,46
2017876254,25
2017876255,36
2017876256,100
2017876257,29
2017876258,79
2017876259,91
2017876260,73
2017876261,88
2017876262,60
2017876263,96
2017876264,32
2017876265,86
2017876266,88
2017876267,94
2017876268,96
2017876269,93
2017876270,34
2017876271,97
2017876272,33
2017876273,82
2017876274,33
2017876275,15
2017876276,58
2017876277,27
2017876278,60
2017876279,64
2017876280,88
- 1先写writable接口:
分析:把学号和成绩都写进writable里面,并且是学号和成绩一起作为key值,类型都是IntWritable类型。
package SequenceScoreAndSno;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.WritableComparable;
public class MyWritable implements WritableComparable<MyWritable>{
private int sno;
private int score;
public int getSno() {
return sno;
}
public void setSno(int sno) {
this.sno = sno;
}
public int getScore() {
return score;
}
@Override
public String toString() {
return "sno=" + sno + ", score=" + score ;
}
public void setScore(int score) {
this.score = score;
}
public MyWritable(){
<