运行hadoop基准测试

最新推荐文章于 2024-07-25 16:38:33 发布

azhao_dn

最新推荐文章于 2024-07-25 16:38:33 发布

阅读量1.1w

点赞数 3

分类专栏： hadoop benchmark 基准测试文章标签： hadoop 测试 jar input output random

本文链接：https://blog.csdn.net/azhao_dn/article/details/6930909

版权

本文介绍了如何使用Hadoop自带的测试用例进行集群性能测试，包括DFSCIOTest、teragen/terasort/teravalidate和nnbench等，详细阐述了各个测试的用途和运行参数，为优化Hadoop集群提供了基准参考。

摘要由CSDN通过智能技术生成

由于需要为hadoop集群采购新的服务器，需要对服务器在hadoop环境下的性能进行测试，所以特地整理了一下hadoop集群自带的测试用例：

bin/hadoop jar hadoop-*test*.jar

运行上述命令，可以得到hadoop-*test*.jar自带的测试程序

An example program must be given as the first argument.
Valid program names are:
  DFSCIOTest: Distributed i/o benchmark of libhdfs.
  DistributedFSCheck: Distributed checkup of the file system consistency.
  MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
  TestDFSIO: Distributed i/o benchmark.
  dfsthroughput: measure hdfs throughput
  filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
  loadgen: Generic map/reduce load generator
  mapredtest: A map/reduce test check.
  mrbench: A map/reduce benchmark that can create many small jobs
  nnbench: A benchmark that stresses the namenode.
  testarrayfile: A test for flat files of binary key/value pairs.
  testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
  testfilesystem: A test for FileSystem read/write.
  testipc: A test for ipc.
  testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
  testrpc: A test for rpc.
  testsequencefile: A test for flat files of binary key value pairs.
  testsequencefileinputformat: A test for sequence file input format.
  testsetfile: A test for flat files of binary key/value pairs.
  testtextinputformat: A test for text input format.
  threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill

其中最常用到的是DFSCIOTest，DFSCIOTest的命令参数如下：

$ bin/hadoop jar hadoop-*test*.jar TestDFSIO
TestDFSIO.0.0.4
Usage: TestDFSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile resultFileNam