Hadoop--Hadoop基准测试(读/写)

最新推荐文章于 2022-05-04 23:01:27 发布

奈斯Ekko

最新推荐文章于 2022-05-04 23:01:27 发布

阅读量591

点赞数 1

文章标签： hadoop

本文链接：https://blog.csdn.net/weixin_42946624/article/details/104504574

版权

本文详细介绍了Hadoop的基准测试工具，包括TestDFSIO用于测试Hadoop的读写速度，nnbench用于测试NameNode的负载能力，以及mrbench用于检查MapReduce小作业的效率。通过示例展示了如何执行这些测试，并分析了测试结果。

摘要由CSDN通过智能技术生成

Hadoop自带了几个基准测试，本文使用的是hadoop-2.6.0

一、Hadoop Test 的测试

[root@master hadoop-2.6.0]# hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar
An example program must be given as the first argument.
Valid program names are:
DFSCIOTest: Distributed i/o benchmark of libhdfs.
DistributedFSCheck: Distributed checkup of the file system consistency.
JHLogAnalyzer: Job History Log analyzer.
MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
SliveTest: HDFS Stress Test and Live Data Verification.
TestDFSIO: Distributed i/o benchmark.
fail: a job that always fails
filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
largesorter: Large-Sort tester
loadgen: Generic map/reduce load generator
mapredtest: A map/reduce test check.
minicluster: Single process HDFS and MR cluster.
mrbench: A map/reduce benchmark that can create many small jobs
nnbench: A benchmark that stresses the namenode.
sleep: A job that sleeps at each map and reduce task.
testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
testfilesystem: A test for FileSystem read/write.
testmapredsort: A map/reduce program that validates the map-reduce framework’s sort.
testsequencefile: A test for flat files of binary key value pairs.
testsequencefileinputformat: A