MapReduce学习写测试

参考

Effective Testing Strategies for MapReduce Applications:https://stackoverflow.com/questions/27136752/powermock-throws-nosuchmethoderror-setmockname
PowerMock throws NoSuchMethodError (setMockName):https://stackoverflow.com/questions/27136752/powermock-throws-nosuchmethoderror-setmockname
Intellij IDEA run coverage之覆盖率测试:https://www.jianshu.com/p/ca3b91e85fea
Idea maven项目不能新建package和class的解决:https://blog.csdn.net/qq_24949727/article/details/52097838
Hadoop现有测试框架探幽:https://blog.csdn.net/azhao_dn/article/details/8065679
Hadoop Herriot测试框架之旅:https://www.linuxidc.com/Linux/2011-10/44984.htm
@Override is not allowed when implementing interface method:https://stackoverflow.com/questions/15402615/override-is-not-allowed-when-implementing-interface-method

测试方式

本次测试的github链接

mrunit

mrunit 已经被退休了,很多人建议不要再使用这个进行测试
千遍一律的WordCount

public class WordCountMRUnitTest {
    MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
    MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
    ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
    @Before
    public void setup() {
        WordCountMapper mapper = new WordCountMapper();
        WordCountReducer reducer = new WordCountReducer();
        mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
        mapDriver.setMapper(mapper);
        reduceDriver = new ReduceDriver<Text, IntWritable, Text, IntWritable>();
        reduceDriver.setReducer(reducer);
        mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>();
        mapReduceDriver.setMapper(mapper);
        mapReduceDriver.setReducer(reducer);
        Configuration conf = new Configuration();
        // add config here as needed
        mapReduceDriver.setConfiguration(conf);
        reduceDriver.setConfiguration(conf);
        mapDriver.setConfiguration(conf);
    }
    @Test
    public void testMapper()  throws IOException {
        mapDriver.withInput(new LongWritable(1), new Text("cat cat dog"));
        mapDriver.withOutput(new Text("cat"), new IntWritable(1));
        mapDriver.withOutput(new Text("cat"), new IntWritable(1));
        mapDriver.withOutput(new Text("dog"), new IntWritable(1));
        mapDriver.runTest();
    }
    @Test
    public void testReducer() throws IOException {
        List<IntWritable> values = new ArrayList<IntWritable>();
        values.add(new IntWritable(1));
        values.add(new IntWritable(1));
        reduceDriver.withInput(new Text("cat"), values);
        reduceDriver.withOutput(new Text("cat"), new IntWritable(2));
        reduceDriver.runTest();
    }
    @Test
    public void testMapReduce() throws IOException {
        mapReduceDriver.withInput(new LongWritable(1), new Text("cat cat dog"));
        mapReduceDriver.addOutput(new Text("cat"), new IntWritable(2));
        mapReduceDriver.addOutput(new Text("dog"), new IntWritable(1));
        mapReduceDriver.runTest();
    }
}

junit

对map,reduce和Driver分别测试,主要用到了Mockito

public class WordCountMapperTest {
    private WordCountMapper mapper;
    private Mapper.Context context;
    private IntWritable one;

    @Before
    public void init() throws IOException, InterruptedException {
        mapper = new WordCountMapper();
        context = mock(Mapper.Context.class); //
        mapper.word = mock(Text.class);
        one = new IntWritable(1);
    }
    @Test
    public void testSingleWord() throws IOException, InterruptedException {
        mapper.map(new LongWritable(1L), new Text("foo"), context);
        InOrder inOrder = inOrder(mapper.word, context);
        assertCountedOnce(inOrder, "foo");
    }
    @Test
    public void testMultipleWords() throws IOException, InterruptedException {
        mapper.map(new LongWritable(1L), new Text("one two three four"), context);
        InOrder inOrder = inOrder(mapper.word, context, mapper.word, context, mapper.word, context, mapper.word, context);
        assertCountedOnce(inOrder, "one");
        assertCountedOnce(inOrder, "two");
        assertCountedOnce(inOrder, "three");
        assertCountedOnce(inOrder, "four");
    }
    private void assertCountedOnce(InOrder inOrder, String w) throws IOException, InterruptedException {
        inOrder.verify(mapper.word).set(eq(w));
        inOrder.verify(context).write(eq(mapper.word), eq(one));
    }
}

public class WordCountReducerTest {
    private WordCountReducer reducer;
    private Reducer.Context context;
    @Before
    public void init() throws IOException, InterruptedException {
        reducer = new WordCountReducer();
        context = mock(Reducer.Context.class);
    }
    @Test
    public void testSingleWord() throws IOException, InterruptedException {
        List<IntWritable> values = Arrays.asList(new IntWritable(1), new IntWritable(4), new IntWritable(7));
        reducer.reduce(new Text("foo"), values, context);
        verify(context).write(new Text("foo"), new IntWritable(12));
    }
}

public class WordCountDriverTest {
    private Configuration conf;
    private Path input;
    private Path output;
    private FileSystem fs;
    @Before
    public void setup() throws IOException {
        conf = new Configuration();
        conf.set("fs.default.name", "file:///");
        conf.set("mapred.job.tracker", "local");
        input = new Path("src/test/resources/input");
        output = new Path("target/output");
        fs = FileSystem.getLocal(conf);
        fs.delete(output, true);
    }
    @Test
    public void test() throws Exception {
        WordCount wordCount = new WordCount();
        wordCount.setConf(conf);
        int exitCode = wordCount.run(new String[] {input.toString(), output.toString()});
        assertEquals(0, exitCode);
        validateOuput();
    }
    private void validateOuput() throws IOException {
        InputStream in = null;
        try {
            in = fs.open(new Path("target/output/part-r-00000"));
            BufferedReader br = new BufferedReader(new InputStreamReader(in));
            assertEquals("five\t1", br.readLine());
            assertEquals("four\t1", br.readLine());
            assertEquals("one\t3", br.readLine());
            assertEquals("six\t1", br.readLine());
            assertEquals("three\t1", br.readLine());
            assertEquals("two\t2", br.readLine());
        } finally {
            IOUtils.closeStream(in);
        }
    }
}

在hadoop本身测试基础上

hadoop说在0.21版本之前,利用MiniDFSCluster模拟hdfs集群环境,看源码可以看到很多例子,比如TeraSort的tests;
之后,hadoop发行版中引入了一个新的测试框架,Large-Scale Automated Test Framework,该框架跟以前的测试框架不同之处在于,基于它之上的测试的开发是基于真正的集群环境的系统层面的,取名叫做Herriot。

将TeraSort的代码和相关测试抽出来尝试其测试过程,过程中发现很多类通过maven引不进来,比如MiniDFSCluster,后来就直接导入相关jar包:hadoop-hdfs-2.8.3-tests.jar,该jar包在安装的hadoop目录的share文件中可以找到。

package TeraSort;

import java.io.File;
import java.io.IOException;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapred.FileAlreadyExistsException;
import org.apache.hadoop.mapred.HadoopTestCase;
import org.apache.hadoop.util.ToolRunner;
import org.junit.After;
import org.junit.Test;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;

public class TestTeraSort extends HadoopTestCase {
    private static Log LOG = LogFactory.getLog(TestTeraSort.class);

    public TestTeraSort()
            throws IOException {
        super(LOCAL_MR, LOCAL_FS, 1, 1);
    }

    @After
    public void tearDown() throws Exception {
        getFileSystem().delete(TEST_DIR, true);
        super.tearDown();
    }

    // Input/Output paths for sort
    private static final Path TEST_DIR = new Path(new File(
            System.getProperty("test.build.data", "/tmp"), "terasort")
            .getAbsoluteFile().toURI().toString());
    private static final Path SORT_INPUT_PATH = new Path(TEST_DIR, "sortin");
    private static final Path SORT_OUTPUT_PATH = new Path(TEST_DIR, "sortout");
    private static final Path TERA_OUTPUT_PATH = new Path(TEST_DIR, "validate");
    private static final String NUM_ROWS = "100";

    private void runTeraGen(Configuration conf, Path sortInput) throws Exception {
        String[] genArgs = {NUM_ROWS, sortInput.toString()};
        // Run TeraGen
        assertEquals(ToolRunner.run(conf, new TeraGen(), genArgs), 0);
    }

    private void runTeraSort(Configuration conf, Path sortInput, Path sortOutput) throws Exception {
        // Setup command-line arguments to 'sort'
        String[] sortArgs = {sortInput.toString(), sortOutput.toString()};

        // Run Sort
        assertEquals(ToolRunner.run(conf, new TeraSort(), sortArgs), 0);
    }

    private void runTeraValidator(Configuration job,Path sortOutput, Path valOutput) throws Exception {
        String[] svArgs = {sortOutput.toString(), valOutput.toString()};

        // Run Tera-Validator
        assertEquals(ToolRunner.run(job, new TeraValidate(), svArgs), 0);
    }

    @Test
    public void testTeraSort() throws Exception {
        // Run TeraGen to generate input for 'terasort'
        runTeraGen(createJobConf(), SORT_INPUT_PATH);

        // Run teragen again to check for FAE
        try {
            runTeraGen(createJobConf(), SORT_INPUT_PATH);
            fail("Teragen output overwritten!");
        } catch (FileAlreadyExistsException fae) {
            LOG.info("Expected exception: ", fae);
        }

        // Run terasort
        runTeraSort(createJobConf(), SORT_INPUT_PATH, SORT_OUTPUT_PATH);

        // Run terasort again to check for FAE
        try {
            runTeraSort(createJobConf(), SORT_INPUT_PATH, SORT_OUTPUT_PATH);
            fail("Terasort output overwritten!");
        } catch (FileAlreadyExistsException fae) {
            LOG.info("Expected exception: ", fae);
        }

        // Run tera-validator to check if sort worked correctly
        runTeraValidator(createJobConf(), SORT_OUTPUT_PATH,
                TERA_OUTPUT_PATH);
    }

    @Test
    public void testTeraSortWithLessThanTwoArgs() throws Exception {
        String[] args = new String[1];
        assertEquals(new TeraSort().run(args), 2);
    }

}

测试率覆盖

idea本身自带有测试覆盖率的,run test时选择run “*Test*” with Coverage

学习过程中遇到的问题

编码错误

@Override is not allowed when implementing interface method 

Project Structure -> Modules -> Language level 选择6

powermockito和mockito版本问题

错误1:java.lang.NoSuchMethodError: 
org.mockito.internal.creation.MockSettingsImpl.setMockName(Lorg/mockito/mock/MockName;)Lorg/mockito/internal/creation/settings/CreationSettings;
错误2:java.lang.NoSuchMethodError: 
org.powermock.reflect.internal.WhiteboxImpl.getOriginalUnmockedType(Ljava/lang/Class;)Ljava/lang/Class;

powermockito和mockito版本需要对应

Mockito PowerMock
2.0.0-beta - 2.0.42-beta 1.6.5+
1.10.19 1.6.4
1.10.8 - 1.10.x 1.6.2+
1.9.5-rc1 - 1.9.5 1.5.0 - 1.5.6
1.9.0-rc1 & 1.9.0 1.4.10 - 1.4.12
1.8.5 1.3.9 - 1.4.9
1.8.4 1.3.7 & 1.3.8
1.8.3 1.3.6
1.8.1 & 1.8.2 1.3.5
1.8 1.3
1.7 1.2.5

自己建的目录不能new java类

点击该目录,右键 -> Mark Directory as -> Sources Root

阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页