转载：运行hadoop基准测试_hadoop-test.jar-CSDN博客

原始链接：https://blog.csdn.net/azhao_dn/article/details/6930909

运行hadoop基准测试

由于需要为hadoop集群采购新的服务器，需要对服务器在hadoop环境下的性能进行测试，所以特地整理了一下hadoop集群自带的测试用例：

```
bin/hadoop jar hadoop-*test*.jar
```
运行上述命令，可以得到hadoop-*test*.jar自带的测试程序
```
 
```
1. An example program must be given as the first argument.
2. Valid program names are:
3. DFSCIOTest: Distributed i/o benchmark of libhdfs.
4. DistributedFSCheck: Distributed checkup of the file system consistency.
5. MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
6. TestDFSIO: Distributed i/o benchmark.
7. dfsthroughput: measure hdfs throughput
8. filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
9. loadgen: Generic map/reduce load generator
10. mapredtest: A map/reduce test check.
11. mrbench: A map/reduce benchmark that can create many small jobs
12. nnbench: A benchmark that stresses the namenode.
13. testarrayfile: A test for flat files of binary key/value pairs.
14. testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
15. testfilesystem: A test for FileSystem read/write.
16. testipc: A test for ipc.
17. testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
18. testrpc: A test for rpc.
19. testsequencefile: A test for flat files of binary key value pairs.
20. testsequencefileinputformat: A test for sequence file input format.
21. testsetfile: A test for flat files of binary key/value pairs.
22. testtextinputformat: A test for text input format.
23. threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill
其中最常用到的是DFSCIOTest，DFSCIOTest的命令参数如下：
```
 
```
1. $ bin/hadoop jar hadoop-*test*.jar TestDFSIO
2. TestDFSIO.0.0.4
3. Usage: TestDFSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile resultFileName] [-bufferSize Bytes]
```
hadoop jar hadoop-*test*.jar TestDFSIO -write -nrFiles 10 -fileSize 1000                                                                           
```
```
hadoop jar hadoop-*test*.jar TestDFSIO -read -nrFiles 10 -fileSize 1000
```
```
hadoop jar hadoop-*test*.jar TestDFSIO -clean
```
```
bin/hadoop jar hadoop-*examples*.jar
```
运行上述命令，可以得到hadoop-*example*.jar自带的测试程序
```
 
```
1. An example program must be given as the first argument.
2. Valid program names are:
3. aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
4. aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
5. dbcount: An example job that count the pageview counts from a database.
6. grep: A map/reduce program that counts the matches of a regex in the input.
7. join: A job that effects a join over sorted, equally partitioned datasets
8. multifilewc: A job that counts words from several files.
9. pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
10. pi: A map/reduce program that estimates Pi using monte-carlo method.
11. randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
12. randomwriter: A map/reduce program that writes 10GB of random data per node.
13. secondarysort: An example defining a secondary sort to the reduce.
14. sleep: A job that sleeps at each map and reduce task.
15. sort: A map/reduce program that sorts the data written by the random writer.
16. sudoku: A sudoku solver.
17. teragen: Generate data for the terasort
18. terasort: Run the terasort
19. teravalidate: Checking results of terasort
20. wordcount: A map/reduce program that counts the words in the input files.
其中最常用的是teragen/terasort/teravalidate，一个完整的terasort测试由三个步骤组成：1）teragen产生数据；2）terasort执行排序；3）teravalidate验证排序结果。其运行命令参数如下：
```
hadoop jar hadoop-*examples*.jar teragen <number of 100-byte rows> <output dir>
```
```
hadoop jar hadoop-*examples*.jar terasort <input dir> <output dir>
```
```
hadoop jar hadoop-*examples*.jar teravalidate <terasort output dir (= input data)> <teravalidate output dir>
```
teravalidate执行验证操作时会输出排序错误的key，当输出结果为空时，表示排序正确
NameNode基准测试nnbench
```
 
```
1. $ bin/hadoop jar hadoop-*test*.jar nnbench
2. NameNode Benchmark 0.4
3. Usage: nnbench <options>
4. Options:
5. -operation <Available operations are create_write open_read rename delete. This option is mandatory>
6. * NOTE: The open_read, rename and delete operations assume that the files they operate on, are already available. The create_write operation must be run before running the other operations.
7. -maps <number of maps. default is 1. This is not mandatory>
8. -reduces <number of reduces. default is 1. This is not mandatory>
9. -startTime <time to start, given in seconds from the epoch. Make sure this is far enough into the future, so all maps (operations) will start at the same time>. default is launch time + 2 mins. This is not mandatory
10. -blockSize <Block size in bytes. default is 1. This is not mandatory>
11. -bytesToWrite <Bytes to write. default is 0. This is not mandatory>
12. -bytesPerChecksum <Bytes per checksum for the files. default is 1. This is not mandatory>
13. -numberOfFiles <number of files to create. default is 1. This is not mandatory>
14. -replicationFactorPerFile <Replication factor for the files. default is 1. This is not mandatory>
15. -baseDir <base DFS path. default is /becnhmarks/NNBench. This is not mandatory>
16. -readFileAfterOpen <true or false. if true, it reads the file and reports the average time to read. This is valid with the open_read operation. default is false. This is not mandatory>
17. -help: Display the help statement
运行案例：
```
$ hadoop jar hadoop-*test*.jar nnbench -operation create_write \
    -maps 12 -reduces 6 -blockSize 1 -bytesToWrite 0 -numberOfFiles 1000 \
    -replicationFactorPerFile 3 -readFileAfterOpen true \
    -baseDir /benchmarks/NNBench-`hostname -s`
```
MapRed基准测试mrbench
```
 
```
1. bin/hadoop jar hadoop-*test*.jar nnbench --help
2. NameNode Benchmark 0.4
3. Usage: nnbench <options>
4. Options:
5. -operation <Available operations are create_write open_read rename delete. This option is mandatory>
6. * NOTE: The open_read, rename and delete operations assume that the files they operate on, are already available. The create_write operation must be run before running the other operations.
7. -maps <number of maps. default is 1. This is not mandatory>
8. -reduces <number of reduces. default is 1. This is not mandatory>
9. -startTime <time to start, given in seconds from the epoch. Make sure this is far enough into the future, so all maps (operations) will start at the same time>. default is launch time + 2 mins. This is not mandatory
10. -blockSize <Block size in bytes. default is 1. This is not mandatory>
11. -bytesToWrite <Bytes to write. default is 0. This is not mandatory>
12. -bytesPerChecksum <Bytes per checksum for the files. default is 1. This is not mandatory>
13. -numberOfFiles <number of files to create. default is 1. This is not mandatory>
14. -replicationFactorPerFile <Replication factor for the files. default is 1. This is not mandatory>
15. -baseDir <base DFS path. default is /becnhmarks/NNBench. This is not mandatory>
16. -readFileAfterOpen <true or false. if true, it reads the file and reports the average time to read. This is valid with the open_read operation. default is false. This is not mandatory>
17. -help: Display the help statement
gridmix测试：gridmix测试是将hadoop自带基准测试进一步打包，一次运行所有测试
```
1）编译：
```
```
 
```
1. cd src/benchmarks/gridmix2
2. ant
2）修改配置文件：vi gridmix-env-2
```
 
```
1. export HADOOP_INSTALL_HOME=/home/test/hadoop
2. export HADOOP_VERSION=hadoop-0.20.203.0
3. export HADOOP_HOME=${HADOOP_INSTALL_HOME}/${HADOOP_VERSION}
4. export HADOOP_CONF_DIR=$HADOOP_HOME/conf
5. export USE_REAL_DATASET=
7. export APP_JAR=${HADOOP_HOME}/hadoop-core-0.20.203.0.jar
8. export EXAMPLE_JAR=${HADOOP_HOME}/hadoop-examples-0.20.203.0.jar
9. export STREAMING_JAR=${HADOOP_HOME}/contrib/streaming/hadoop-streaming-0.20.203.0.jar
```
3）产生测试数据：sh generateGridmix2data.sh
```
4）运行测试：
```
 
```
1. $ chmod +x rungridmix_2
2. $ ./rungridmix_2
```
 
```
参考资料：
```
 
```