测试Hadoop程序

MRUnit

MRUnit是一个能够然测试Mapper、Reducer、和MapReduce程序的集成测试库

安装MRUnit：
在Maven的POM.XML文件中添加如下依赖

<dependency>
     <groupId>junit</groupId>
     <artifactId>junit</artifactId>
     <version>4.12</version>
</dependency>
<dependency>
     <groupId>org.apache.mrunit</groupId>
     <artifactId>mrunit</artifactId>
     <version>1.1.0</version>
     <classifier>hadoop2</classifier>
 </dependency>

LocalJobRunner

MapReduce程序可以使用LocalJobRunner进行从头到尾的测试

但是它是单线程的，只能运行一个Reducer

MiniMRCluster

MiniMRCuster使用的Jar文件：

• hadoop-mapreduce-client-jobclient-版本号-tests.jar
• hadoop-common-版本号-tests.jar
• hadoop-hdfs-版本号-tests.jar
• hadoop-yarn-server-tests-版本号-tests.jar
• hadoop-yarn-server-nodemanager-版本号.jar
• hadoop-mapreduce-client-hs-版本号.jar

在Maven的POM.XML中添加如下测试依赖：

<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
            <version>2.7.3</version>
            <classifier>tests</classifier>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.7.3</version>
            <classifier>tests</classifier>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.7.3</version>
            <classifier>tests</classifier>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-server-tests</artifactId>
            <version>2.7.3</version>
            <classifier>tests</classifier>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-server-nodemanager</artifactId>
            <version>2.7.3</version>
            <classifier>tests</classifier>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-hs</artifactId>
            <version>2.7.3</version>
            <classifier>tests</classifier>
        </dependency>

局限性，它是一个正真模拟多线程的MapReduce框架，在这种方法下内存不足是一个真正的风险。