首先下载MRUnitjar包,地址:https://repository.apache.org/content/repositories/releases/org/apache/mrunit/mrunit/,选择和你使用的hadoop对应版本,将jar包导入项目。
MRUnit的使用很简单,流程如下:
1:根据业务要求编写Map类,Reduce类
2:编写测试类。
3:运行测试,得到结果。
准备测试数据:
- CDRID;CDRType;Phone1;Phone2;SMS Status Code
655209;1;796764372490213;804422938115889;6
353415;0;356857119806206;287572231184798;4
835699;1;252280313968413;889717902341635;0
业务要求:
找出CDRType是1的,所有的SMS Status Code的类型汇总。
map结果应该输出如下信息:
6, 1
0, 1
下面就是准备map,reduce类。
Map 类:
public
class
SMSCDRMapper
extends
Mapper<LongWritable, Text, Text, IntWritable> {
private
Text status =
new
Text();
private
final
static
IntWritable addOne =
new
IntWritable(
1
);
/**
* Returns the SMS status code and its count
*/
protected
void
map(LongWritable key, Text value, Context context)
throws
java.io.IOException, InterruptedException {
//655209;1;796764372490213;804422938115889;6 is the Sample record format
String[] line = value.toString().split(
";"
);
// If record is of SMS CDR
if
(Integer.parseInt(line[
1
]) ==
1
) {
status.set(line[
4
]);
context.write(status, addOne);
}
}
}
Reduce 类:
public
class
SMSCDRReducer
extends
Reducer<Text, IntWritable, Text, IntWritable> {
protected
void
reduce(Text key, Iterable<IntWritable> values, Context context)
throws
java.io.IOException, InterruptedException {
int
sum =
0
;
for
(IntWritable value : values) {
sum += value.get();
}
context.write(key,
new
IntWritable(sum));
}
}
测试Test类:
import
java.util.ArrayList;
import
java.util.List;
import
org.apache.hadoop.io.IntWritable;
import
org.apache.hadoop.io.LongWritable;
import
org.apache.hadoop.io.Text;
import
org.apache.hadoop.mrunit.mapreduce.MapDriver;
import
org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
import
org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
import
org.junit.Before;
import
org.junit.Test;
public
class
SMSCDRMapperReducerTest {
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
@Before
public
void
setUp() {
SMSCDRMapper mapper =
new
SMSCDRMapper();
SMSCDRReducer reducer =
new
SMSCDRReducer();
mapDriver = MapDriver.newMapDriver(mapper);
reduceDriver = ReduceDriver.newReduceDriver(reducer);
mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
}
@Test
public
void
testMapper() {
mapDriver.withInput(
new
LongWritable(),
new
Text(
"655209;1;796764372490213;804422938115889;6"
));
mapDriver.withOutput(
new
Text(
"6"
),
new
IntWritable(
1
));
mapDriver.runTest();
}
@Test
public
void
testReducer() {
List<IntWritable> values =
new
ArrayList<IntWritable>();
values.add(
new
IntWritable(
1
));
values.add(
new
IntWritable(
1
));
reduceDriver.withInput(
new
Text(
"6"
), values);
reduceDriver.withOutput(
new
Text(
"6"
),
new
IntWritable(
2
));
reduceDriver.runTest();
}
@Test
public
void
testMapReduce() {
mapReduceDriver.withInput(
new
LongWritable(),
new
Text(
"655209;1;796764372490213;804422938115889;6"
));
List<IntWritable> values =
new
ArrayList<IntWritable>();
values.add(
new
IntWritable(
1
));
values.add(
new
IntWritable(
1
));
mapReduceDriver.withOutput(
new
Text(
"6"
),
new
IntWritable(
2
));
mapReduceDriver.runTest();
}
}
最后通过Junit的方式调用运行就可以了。这样就可以实现了在本地没有集群环境的方式下快速方便的进行MR功能测试。