hadoop测试

Hadoop基准测试
Hadoop本身自带的一些基准测试的jar包
进入hadoop的安装目录，可以看到有一个hadoop-mapreduce-examples-xxx.jar的jar包，直接不给参数直接运行,会列出所有的测试程序
[root@bigdata-server-1hadoop]#
hadoop jar hadoop-mapreduce-examples-2.7.1.2.3.4.0.jar
An example program must be given as the first argument.
Valid program names are:
aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
dbcount: An example job that count the pageview counts from a database.
distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
grep: A map/reduce program that counts the matches of a regex in the input.
join: A job that effects a join over sorted, equally partitioned datasets
multifilewc: A job that counts words from several files.
pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
randomwriter: A map/reduce program that writes 10GB of random data per node.
secondarysort: An example defining a secondary sort to the reduce.
sort: A map/reduce program that sorts the data written by the random writer.
sudoku: A sudoku solver.
teragen: Generate data for the terasort
terasort: Run the terasort
teravalidate: Checking results of terasort
wordcount: A map/reduce program that counts the words in the input files.
wordmean: A map/reduce program that counts the average length of the words in the input files.
wordmedian: A map/reduce program that counts the median length of the words in the input files.
wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

TestDFSIO write(测试写入的能力)
TestDFSIO用于测试HDFS的IO性能，使用一个MapReduce作业来并行执行读写操作，每个map任务用于读或写每个文件，map的输出用于收集与处理文件相关的统计信息，reduce用于积累统计信息，并产生summary。TestDFSIO例子：
Hadoop jar ./hadoop-mapreduce-client-jobclient-2.7.1-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 1000MB
清空数据
hadoop jar ./hadoop-mapreduce-client-jobclient-2.7.1-tests.jar TestDFSIO -clean
TestDFSIO read(测试读取的能力)
Hadoop jar ./hadoop-mapreduce-client-jobclient-2.7.1-tests.jar TestDFSIO -read -nrFIles 10 -filesize 1000