基于HDP2.6.0.3-8的Hadoop TestDFSIO、mrbench和nnbench是三个广泛被使用的测试

1、Hadoop Test 的测试
# cd  /usr/hdp/2.6.0.3-8/hadoop-mapreduce


查看参数  
# hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar 
An example program must be given as the first argument.
Valid program names are:
  DFSCIOTest: Distributed i/o benchmark of libhdfs.
  DistributedFSCheck: Distributed checkup of the file system consistency.
  JHLogAnalyzer: Job History Log analyzer.
  MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
  NNdataGenerator: Generate the data to be used by NNloadGenerator
  NNloadGenerator: Generate load on Namenode using NN loadgenerator run WITHOUT MR
  NNloadGeneratorMR: Generate load on Namenode using NN loadgenerator run as MR job
  NNstructureGenerator: Generate the structure to be used by NNdataGenerator
  SliveTest: HDFS Stress Test and Live Data Verification.
  TestDFSIO: Distributed i/o benchmark.
  fail: a job that always fails
  filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
  largesorter: Large-Sort tester
  loadgen: Generic map/reduce load generator
  mapredtest: A map/reduce test check.
  minicluster: Single process HDFS and MR cluster.
  mrbench: A map/reduce benchmark that can create many small jobs
  nnbench: A benchmark that stresses the namenode.
  sleep: A job that sleeps at each map and reduce task.
  testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
  testfilesystem: A test for FileSystem read/write.
  testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
  testsequencefile: A test for flat files of binary key value pairs.
  testsequencefileinputformat: A test for sequence file input format.
  testtextinputformat: A test for text input format.
  threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill


2、TestDFSIO write的性能测试
测试hadoop写的速度


查看参数
# hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar TestDFSIO
17/11/21 14:46:38 INFO fs.TestDFSIO: TestDFSIO.1.8
Missing arguments.
Usage: TestDFSIO [genericOptions] -read [-random | -backward | -skip [-skipSize Size]] | -write | -append | -truncate | -clean [-compression codecClassName] [-nrFiles N] [-size Size[B|KB|MB|GB|TB]] [-resFile resultFileName] [-bufferSize Bytes] [-rootDir]


运行实例
#  hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar TestDFSIO -write -nrFiles 10 -size 10MB
17/11/21 14:52:39 INFO fs.TestDFSIO: TestDFSIO.1.8
17/11/21 14:52:39 INFO fs.TestDFSIO: nrFiles = 10
17/11/21 14:52:39 INFO fs.TestDFSIO: nrBytes (MB) = 10.0
17/11/21 14:52:39 INFO fs.TestDFSIO: bufferSize = 1000000
17/11/21 14:52:39 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
17/11/21 14:52:40 INFO fs.TestDFSIO: creating control file: 10485760 bytes, 10 files
17/11/21 14:52:41 INFO fs.TestDFSIO: created control files for: 10 files
17/11/21 14:52:41 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 14:52:41 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 14:52:42 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 14:52:42 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 14:52:42 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 175 for admin on 192.168.3.130:8020
17/11/21 14:52:42 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 175 for admin)
17/11/21 14:52:42 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511247160933, maxDate=1511851960933, sequenceNumber=3, masterKeyId=3)
17/11/21 14:52:42 INFO mapred.FileInputFormat: Total input paths to process : 10
17/11/21 14:52:43 INFO mapreduce.JobSubmitter: number of splits:10
17/11/21 14:52:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1511147390118_0001
17/11/21 14:52:43 INFO mapreduce.JobSubmitter: Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511247160933, maxDate=1511851960933, sequenceNumber=3, masterKeyId=3)
17/11/21 14:52:43 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 175 for admin)
17/11/21 14:52:44 INFO impl.TimelineClientImpl: Timeline service address: http://ict-nn-01.cars.com:8188/ws/v1/timeline/
17/11/21 14:52:45 INFO impl.YarnClientImpl: Submitted application application_1511147390118_0001
17/11/21 14:52:45 INFO mapreduce.Job: The url to track the job: http://ict-nn-01.cars.com:8088/proxy/application_1511147390118_0001/
17/11/21 14:52:45 INFO mapreduce.Job: Running job: job_1511147390118_0001
17/11/21 14:53:22 INFO mapreduce.Job: Job job_1511147390118_0001 running in uber mode : false
17/11/21 14:53:22 INFO mapreduce.Job:  map 0% reduce 0%
17/11/21 14:53:31 INFO mapreduce.Job:  map 30% reduce 0%
17/11/21 14:53:35 INFO mapreduce.Job:  map 40% reduce 0%
17/11/21 14:53:36 INFO mapreduce.Job:  map 50% reduce 0%
17/11/21 14:53:38 INFO mapreduce.Job:  map 80% reduce 0%
17/11/21 14:53:40 INFO mapreduce.Job:  map 100% reduce 0%
17/11/21 14:53:44 INFO mapreduce.Job:  map 100% reduce 100%
17/11/21 14:53:44 INFO mapreduce.Job: Job job_1511147390118_0001 completed successfully
17/11/21 14:53:44 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=829
                FILE: Number of bytes written=1698594
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=2440
                HDFS: Number of bytes written=104857676
                HDFS: Number of read operations=43
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=12
        Job Counters 
                Launched map tasks=10
                Launched reduce tasks=1
                Data-local map tasks=10
                Total time spent by all maps in occupied slots (ms)=139952
                Total time spent by all reduces in occupied slots (ms)=20196
                Total time spent by all map tasks (ms)=69976
                Total time spent by all reduce tasks (ms)=5049
                Total vcore-milliseconds taken by all map tasks=69976
                Total vcore-milliseconds taken by all reduce tasks=5049
                Total megabyte-milliseconds taken by all map tasks=143310848
                Total megabyte-milliseconds taken by all reduce tasks=20680704
        Map-Reduce Framework
                Map input records=10
                Map output records=50
                Map output bytes=723
                Map output materialized bytes=883
                Input split bytes=1320
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=883
                Reduce input records=50
                Reduce output records=5
                Spilled Records=100
                Shuffled Maps =10
                Failed Shuffles=0
                Merged Map outputs=10
                GC time elapsed (ms)=1421
                CPU time spent (ms)=31750
                Physical memory (bytes) snapshot=14945345536
                Virtual memory (bytes) snapshot=42306801664
                Total committed heap usage (bytes)=13905166336
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=1120
        File Output Format Counters 
                Bytes Written=76
17/11/21 14:53:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
17/11/21 14:53:44 INFO fs.TestDFSIO:            Date & time: Tue Nov 21 14:53:44 CST 2017
17/11/21 14:53:44 INFO fs.TestDFSIO:        Number of files: 10
17/11/21 14:53:44 INFO fs.TestDFSIO: Total MBytes processed: 100.0
17/11/21 14:53:44 INFO fs.TestDFSIO:      Throughput mb/sec: 19.485580670303975
17/11/21 14:53:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 24.091276168823242
17/11/21 14:53:44 INFO fs.TestDFSIO:  IO rate std deviation: 9.242316274402379
17/11/21 14:53:44 INFO fs.TestDFSIO:     Test exec time sec: 63.103
17/11/21 14:53:44 INFO fs.TestDFSIO:




查看数据
# hadoop fs -ls -h  /benchmarks/TestDFSIO/io_data       
Found 10 items
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_0
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_1
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_2
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_3
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_4
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_5
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_6
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_7
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_8
-rw-r--r--   3 admin hdfs       10 M 2017-11-21 14:53 /benchmarks/TestDFSIO/io_data/test_io_9




查看执行的结果
# cat TestDFSIO_results.log 
----- TestDFSIO ----- : write
           Date & time: Tue Nov 21 14:53:44 CST 2017
       Number of files: 10
Total MBytes processed: 100.0
     Throughput mb/sec: 19.485580670303975
Average IO rate mb/sec: 24.091276168823242
 IO rate std deviation: 9.242316274402379
    Test exec time sec: 63.103




3、TestDFSIO read的性能测试
测试hadoop读文件的速度


#  hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar TestDFSIO -read -nrFiles 10 -size 10
17/11/21 15:03:54 INFO fs.TestDFSIO: TestDFSIO.1.8
17/11/21 15:03:54 INFO fs.TestDFSIO: nrFiles = 10
17/11/21 15:03:54 INFO fs.TestDFSIO: nrBytes (MB) = 10.0
17/11/21 15:03:54 INFO fs.TestDFSIO: bufferSize = 1000000
17/11/21 15:03:54 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
17/11/21 15:03:56 INFO fs.TestDFSIO: creating control file: 10485760 bytes, 10 files
17/11/21 15:03:57 INFO fs.TestDFSIO: created control files for: 10 files
17/11/21 15:03:57 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:03:57 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:03:57 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:03:57 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:03:57 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 176 for admin on 192.168.3.130:8020
17/11/21 15:03:57 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 176 for admin)
17/11/21 15:03:57 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511247836488, maxDate=1511852636488, sequenceNumber=4, masterKeyId=3)
17/11/21 15:03:58 INFO mapred.FileInputFormat: Total input paths to process : 10
17/11/21 15:03:58 INFO mapreduce.JobSubmitter: number of splits:10
17/11/21 15:03:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1511147390118_0002
17/11/21 15:03:58 INFO mapreduce.JobSubmitter: Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511247836488, maxDate=1511852636488, sequenceNumber=4, masterKeyId=3)
17/11/21 15:03:58 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 176 for admin)
17/11/21 15:03:59 INFO impl.TimelineClientImpl: Timeline service address: http://ict-nn-01.cars.com:8188/ws/v1/timeline/
17/11/21 15:04:00 INFO impl.YarnClientImpl: Submitted application application_1511147390118_0002
17/11/21 15:04:00 INFO mapreduce.Job: The url to track the job: http://ict-nn-01.cars.com:8088/proxy/application_1511147390118_0002/
17/11/21 15:04:00 INFO mapreduce.Job: Running job: job_1511147390118_0002
17/11/21 15:04:12 INFO mapreduce.Job: Job job_1511147390118_0002 running in uber mode : false
17/11/21 15:04:12 INFO mapreduce.Job:  map 0% reduce 0%
17/11/21 15:04:18 INFO mapreduce.Job:  map 20% reduce 0%
17/11/21 15:04:23 INFO mapreduce.Job:  map 50% reduce 0%
17/11/21 15:04:24 INFO mapreduce.Job:  map 70% reduce 0%
17/11/21 15:04:29 INFO mapreduce.Job:  map 90% reduce 0%
17/11/21 15:04:30 INFO mapreduce.Job:  map 100% reduce 0%
17/11/21 15:04:33 INFO mapreduce.Job:  map 100% reduce 100%
17/11/21 15:04:33 INFO mapreduce.Job: Job job_1511147390118_0002 completed successfully
17/11/21 15:04:33 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=833
                FILE: Number of bytes written=1698547
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=104860040
                HDFS: Number of bytes written=77
                HDFS: Number of read operations=53
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=10
                Launched reduce tasks=1
                Data-local map tasks=10
                Total time spent by all maps in occupied slots (ms)=117680
                Total time spent by all reduces in occupied slots (ms)=13048
                Total time spent by all map tasks (ms)=58840
                Total time spent by all reduce tasks (ms)=3262
                Total vcore-milliseconds taken by all map tasks=58840
                Total vcore-milliseconds taken by all reduce tasks=3262
                Total megabyte-milliseconds taken by all map tasks=120504320
                Total megabyte-milliseconds taken by all reduce tasks=13361152
        Map-Reduce Framework
                Map input records=10
                Map output records=50
                Map output bytes=727
                Map output materialized bytes=887
                Input split bytes=1320
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=887
                Reduce input records=50
                Reduce output records=5
                Spilled Records=100
                Shuffled Maps =10
                Failed Shuffles=0
                Merged Map outputs=10
                GC time elapsed (ms)=1367
                CPU time spent (ms)=11580
                Physical memory (bytes) snapshot=14952542208
                Virtual memory (bytes) snapshot=42249330688
                Total committed heap usage (bytes)=14047772672
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=1120
        File Output Format Counters 
                Bytes Written=77
17/11/21 15:04:33 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
17/11/21 15:04:33 INFO fs.TestDFSIO:            Date & time: Tue Nov 21 15:04:33 CST 2017
17/11/21 15:04:33 INFO fs.TestDFSIO:        Number of files: 10
17/11/21 15:04:33 INFO fs.TestDFSIO: Total MBytes processed: 100.0
17/11/21 15:04:33 INFO fs.TestDFSIO:      Throughput mb/sec: 617.283950617284
17/11/21 15:04:33 INFO fs.TestDFSIO: Average IO rate mb/sec: 688.1331176757812
17/11/21 15:04:33 INFO fs.TestDFSIO:  IO rate std deviation: 182.42935237458195
17/11/21 15:04:33 INFO fs.TestDFSIO:     Test exec time sec: 36.148
17/11/21 15:04:33 INFO fs.TestDFSIO: 




查看运行的情况
# cat TestDFSIO_results.log 
----- TestDFSIO ----- : write
           Date & time: Tue Nov 21 14:53:44 CST 2017
       Number of files: 10
Total MBytes processed: 100.0
     Throughput mb/sec: 19.485580670303975
Average IO rate mb/sec: 24.091276168823242
 IO rate std deviation: 9.242316274402379
    Test exec time sec: 63.103


----- TestDFSIO ----- : read
           Date & time: Tue Nov 21 15:04:33 CST 2017
       Number of files: 10
Total MBytes processed: 100.0
     Throughput mb/sec: 617.283950617284
Average IO rate mb/sec: 688.1331176757812
 IO rate std deviation: 182.42935237458195
    Test exec time sec: 36.148




4、清空测试数据
#  hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar TestDFSIO -clean
17/11/21 15:15:35 INFO fs.TestDFSIO: TestDFSIO.1.8
17/11/21 15:15:35 INFO fs.TestDFSIO: nrFiles = 1
17/11/21 15:15:35 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
17/11/21 15:15:35 INFO fs.TestDFSIO: bufferSize = 1000000
17/11/21 15:15:35 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
17/11/21 15:15:35 INFO fs.TestDFSIO: Cleaning up test files


5、查看hadoop文件系统
# hadoop fs -ls /benchmarks/




6、nnbench 测试 [NameNode benchmark (nnbench)]
nnbench用于测试NameNode的负载，它会生成很多与HDFS相关的请求，给NameNode施加较大的压力。
这个测试能在HDFS上创建、读取、重命名和删除文件操作


查看nnbench选项
#  hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar nnbench
NameNode Benchmark 0.4
Usage: nnbench <options>
Options:
        -operation <Available operations are create_write open_read rename delete. This option is mandatory>
         * NOTE: The open_read, rename and delete operations assume that the files they operate on, are already available. The create_write operation must be run before running the other operations.
        -maps <number of maps. default is 1. This is not mandatory>
        -reduces <number of reduces. default is 1. This is not mandatory>
        -startTime <time to start, given in seconds from the epoch. Make sure this is far enough into the future, so all maps (operations) will start at the same time. default is launch time + 2 mins. This is not mandatory>
        -blockSize <Block size in bytes. default is 1. This is not mandatory>
        -bytesToWrite <Bytes to write. default is 0. This is not mandatory>
        -bytesPerChecksum <Bytes per checksum for the files. default is 1. This is not mandatory>
        -numberOfFiles <number of files to create. default is 1. This is not mandatory>
        -replicationFactorPerFile <Replication factor for the files. default is 1. This is not mandatory>
        -baseDir <base DFS path. default is /becnhmarks/NNBench. This is not mandatory>
        -readFileAfterOpen <true or false. if true, it reads the file and reports the average time to read. This is valid with the open_read operation. default is false. This is not mandatory>
        -help: Display the help statement


以下例子使用10个mapper和5个reducer来创建1000个文件
#  hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar nnbench -operation create_write -maps 10 -reduces 5 -numberOfFiles 1000 -replicationFactorPerFile 3 -readFileAfterOpen true
NameNode Benchmark 0.4
17/11/21 15:19:08 INFO hdfs.NNBench: Test Inputs: 
17/11/21 15:19:08 INFO hdfs.NNBench:            Test Operation: create_write
17/11/21 15:19:08 INFO hdfs.NNBench:                Start time: 2017-11-21 15:21:08,692
17/11/21 15:19:08 INFO hdfs.NNBench:            Number of maps: 10
17/11/21 15:19:08 INFO hdfs.NNBench:         Number of reduces: 5
17/11/21 15:19:08 INFO hdfs.NNBench:                Block Size: 1
17/11/21 15:19:08 INFO hdfs.NNBench:            Bytes to write: 0
17/11/21 15:19:08 INFO hdfs.NNBench:        Bytes per checksum: 1
17/11/21 15:19:08 INFO hdfs.NNBench:           Number of files: 1000
17/11/21 15:19:08 INFO hdfs.NNBench:        Replication factor: 3
17/11/21 15:19:08 INFO hdfs.NNBench:                  Base dir: /benchmarks/NNBench
17/11/21 15:19:08 INFO hdfs.NNBench:      Read file after open: true
17/11/21 15:19:10 INFO hdfs.NNBench: Deleting data directory
17/11/21 15:19:10 INFO hdfs.NNBench: Creating 10 control files
17/11/21 15:19:11 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:19:11 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:19:11 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:19:11 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:19:11 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 177 for admin on 192.168.3.130:8020
17/11/21 15:19:11 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 177 for admin)
17/11/21 15:19:11 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511248750067, maxDate=1511853550067, sequenceNumber=5, masterKeyId=3)
17/11/21 15:19:11 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/11/21 15:19:12 INFO mapred.FileInputFormat: Total input paths to process : 10
17/11/21 15:19:12 INFO mapreduce.JobSubmitter: number of splits:10
17/11/21 15:19:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1511147390118_0003
17/11/21 15:19:13 INFO mapreduce.JobSubmitter: Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511248750067, maxDate=1511853550067, sequenceNumber=5, masterKeyId=3)
17/11/21 15:19:13 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 177 for admin)
17/11/21 15:19:14 INFO impl.TimelineClientImpl: Timeline service address: http://ict-nn-01.cars.com:8188/ws/v1/timeline/
17/11/21 15:19:14 INFO impl.YarnClientImpl: Submitted application application_1511147390118_0003
17/11/21 15:19:14 INFO mapreduce.Job: The url to track the job: http://ict-nn-01.cars.com:8088/proxy/application_1511147390118_0003/
17/11/21 15:19:14 INFO mapreduce.Job: Running job: job_1511147390118_0003
17/11/21 15:19:37 INFO mapreduce.Job: Job job_1511147390118_0003 running in uber mode : false
17/11/21 15:19:37 INFO mapreduce.Job:  map 0% reduce 0%
17/11/21 15:19:50 INFO mapreduce.Job:  map 33% reduce 0%
17/11/21 15:21:15 INFO mapreduce.Job:  map 50% reduce 0%
17/11/21 15:21:21 INFO mapreduce.Job:  map 90% reduce 0%
17/11/21 15:21:22 INFO mapreduce.Job:  map 100% reduce 0%
17/11/21 15:21:25 INFO mapreduce.Job:  map 100% reduce 20%
17/11/21 15:21:27 INFO mapreduce.Job:  map 100% reduce 40%
17/11/21 15:21:30 INFO mapreduce.Job:  map 100% reduce 60%
17/11/21 15:21:33 INFO mapreduce.Job:  map 100% reduce 80%
17/11/21 15:21:34 INFO mapreduce.Job:  map 100% reduce 100%
17/11/21 15:21:35 INFO mapreduce.Job: Job job_1511147390118_0003 completed successfully
17/11/21 15:21:35 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=1775
                FILE: Number of bytes written=2340320
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=2550
                HDFS: Number of bytes written=182
                HDFS: Number of read operations=55
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=5010
        Job Counters 
                Launched map tasks=10
                Launched reduce tasks=5
                Data-local map tasks=10
                Total time spent by all maps in occupied slots (ms)=1003004
                Total time spent by all reduces in occupied slots (ms)=93784
                Total time spent by all map tasks (ms)=501502
                Total time spent by all reduce tasks (ms)=23446
                Total vcore-milliseconds taken by all map tasks=501502
                Total vcore-milliseconds taken by all reduce tasks=23446
                Total megabyte-milliseconds taken by all map tasks=1027076096
                Total megabyte-milliseconds taken by all reduce tasks=96034816
        Map-Reduce Framework
                Map input records=10
                Map output records=75
                Map output bytes=1595
                Map output materialized bytes=2045
                Input split bytes=1310
                Combine input records=0
                Combine output records=0
                Reduce input groups=8
                Reduce shuffle bytes=2045
                Reduce input records=75
                Reduce output records=8
                Spilled Records=150
                Shuffled Maps =50
                Failed Shuffles=0
                Merged Map outputs=50
                GC time elapsed (ms)=5274
                CPU time spent (ms)=43980
                Physical memory (bytes) snapshot=16229191680
                Virtual memory (bytes) snapshot=64224763904
                Total committed heap usage (bytes)=15164506112
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=1240
        File Output Format Counters 
                Bytes Written=182
17/11/21 15:21:35 INFO hdfs.NNBench: -------------- NNBench -------------- : 
17/11/21 15:21:35 INFO hdfs.NNBench:                                Version: NameNode Benchmark 0.4
17/11/21 15:21:35 INFO hdfs.NNBench:                            Date & time: 2017-11-21 15:21:35,703
17/11/21 15:21:35 INFO hdfs.NNBench: 
17/11/21 15:21:35 INFO hdfs.NNBench:                         Test Operation: create_write
17/11/21 15:21:35 INFO hdfs.NNBench:                             Start time: 2017-11-21 15:21:08,692
17/11/21 15:21:35 INFO hdfs.NNBench:                            Maps to run: 10
17/11/21 15:21:35 INFO hdfs.NNBench:                         Reduces to run: 5
17/11/21 15:21:35 INFO hdfs.NNBench:                     Block Size (bytes): 1
17/11/21 15:21:35 INFO hdfs.NNBench:                         Bytes to write: 0
17/11/21 15:21:35 INFO hdfs.NNBench:                     Bytes per checksum: 1
17/11/21 15:21:35 INFO hdfs.NNBench:                        Number of files: 1000
17/11/21 15:21:35 INFO hdfs.NNBench:                     Replication factor: 3
17/11/21 15:21:35 INFO hdfs.NNBench:             Successful file operations: 0
17/11/21 15:21:35 INFO hdfs.NNBench: 
17/11/21 15:21:35 INFO hdfs.NNBench:         # maps that missed the barrier: 5
17/11/21 15:21:35 INFO hdfs.NNBench:                           # exceptions: 5000
17/11/21 15:21:35 INFO hdfs.NNBench: 
17/11/21 15:21:35 INFO hdfs.NNBench:                TPS: Create/Write/Close: 0
17/11/21 15:21:35 INFO hdfs.NNBench: Avg exec time (ms): Create/Write/Close: Infinity
17/11/21 15:21:35 INFO hdfs.NNBench:             Avg Lat (ms): Create/Write: NaN
17/11/21 15:21:35 INFO hdfs.NNBench:                    Avg Lat (ms): Close: NaN
17/11/21 15:21:35 INFO hdfs.NNBench: 
17/11/21 15:21:35 INFO hdfs.NNBench:                  RAW DATA: AL Total #1: 0
17/11/21 15:21:35 INFO hdfs.NNBench:                  RAW DATA: AL Total #2: 0
17/11/21 15:21:35 INFO hdfs.NNBench:               RAW DATA: TPS Total (ms): 21176
17/11/21 15:21:35 INFO hdfs.NNBench:        RAW DATA: Longest Map Time (ms): 4535.0
17/11/21 15:21:35 INFO hdfs.NNBench:                    RAW DATA: Late maps: 5
17/11/21 15:21:35 INFO hdfs.NNBench:              RAW DATA: # of exceptions: 5000
17/11/21 15:21:35 INFO hdfs.NNBench: 


查看结果
# cat NNBench_results.log 
-------------- NNBench -------------- : 
                               Version: NameNode Benchmark 0.4
                           Date & time: 2017-11-21 15:21:35,703


                        Test Operation: create_write
                            Start time: 2017-11-21 15:21:08,692
                           Maps to run: 10
                        Reduces to run: 5
                    Block Size (bytes): 1
                        Bytes to write: 0
                    Bytes per checksum: 1
                       Number of files: 1000
                    Replication factor: 3
            Successful file operations: 0


        # maps that missed the barrier: 5
                          # exceptions: 5000


               TPS: Create/Write/Close: 0
Avg exec time (ms): Create/Write/Close: Infinity
            Avg Lat (ms): Create/Write: NaN
                   Avg Lat (ms): Close: NaN


                 RAW DATA: AL Total #1: 0
                 RAW DATA: AL Total #2: 0
              RAW DATA: TPS Total (ms): 21176
       RAW DATA: Longest Map Time (ms): 4535.0
                   RAW DATA: Late maps: 5
             RAW DATA: # of exceptions: 5000




7、mrbench测试[MapReduce benchmark (mrbench)]
mrbench会多次重复执行一个小作业，用于检查在机群上小作业的运行是否可重复以及运行是否高效。




查看帮助
#  hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar mrbench --help
MRBenchmark.0.0.2
Usage: mrbench [-baseDir <base DFS path for output/input, default is /benchmarks/MRBench>] [-jar <local path to job jar file containing Mapper and Reducer implementations, default is current jar file>] [-numRuns <number of times to run the job, default is 1>] [-maps <number of maps for each run, default is 2>] [-reduces <number of reduces for each run, default is 1>] [-inputLines <number of input lines to generate, default is 1>] [-inputType <type of input to generate, one of ascending (default), descending, random>] [-verbose]




下面的例子会运行一个小作业2次
#  hadoop jar hadoop-mapreduce-client-jobclient-2.7.3.2.6.0.3-8.jar mrbench -numRuns 2
 MRBenchmark.0.0.2
17/11/21 15:30:40 INFO mapred.MRBench: creating control file: 1 numLines, ASCENDING sortOrder
17/11/21 15:30:40 INFO mapred.MRBench: created control file: /benchmarks/MRBench/mr_input/input_-2146295000.txt
17/11/21 15:30:40 INFO mapred.MRBench: Running job 0: input=hdfs://ict-nn-01.cars.com:8020/benchmarks/MRBench/mr_input output=hdfs://ict-nn-01.cars.com:8020/benchmarks/MRBench/mr_output/output_-627187026
17/11/21 15:30:40 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:30:41 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:30:41 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:30:41 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:30:41 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 179 for admin on 192.168.3.130:8020
17/11/21 15:30:41 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 179 for admin)
17/11/21 15:30:41 INFO security.TokenCache: Got dt for hdfs://ict-nn-01.cars.com:8020; Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511249440159, maxDate=1511854240159, sequenceNumber=7, masterKeyId=3)
17/11/21 15:30:42 INFO mapred.FileInputFormat: Total input paths to process : 1
17/11/21 15:30:42 INFO mapreduce.JobSubmitter: number of splits:2
17/11/21 15:30:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1511147390118_0006
17/11/21 15:30:42 INFO mapreduce.JobSubmitter: Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511249440159, maxDate=1511854240159, sequenceNumber=7, masterKeyId=3)
17/11/21 15:30:42 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 179 for admin)
17/11/21 15:30:43 INFO impl.TimelineClientImpl: Timeline service address: http://ict-nn-01.cars.com:8188/ws/v1/timeline/
17/11/21 15:30:44 INFO impl.YarnClientImpl: Submitted application application_1511147390118_0006
17/11/21 15:30:44 INFO mapreduce.Job: The url to track the job: http://ict-nn-01.cars.com:8088/proxy/application_1511147390118_0006/
17/11/21 15:30:44 INFO mapreduce.Job: Running job: job_1511147390118_0006
17/11/21 15:31:00 INFO mapreduce.Job: Job job_1511147390118_0006 running in uber mode : false
17/11/21 15:31:00 INFO mapreduce.Job:  map 0% reduce 0%
17/11/21 15:31:07 INFO mapreduce.Job:  map 100% reduce 0%
17/11/21 15:31:15 INFO mapreduce.Job:  map 100% reduce 100%
17/11/21 15:31:16 INFO mapreduce.Job: Job job_1511147390118_0006 completed successfully
17/11/21 15:31:16 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=13
                FILE: Number of bytes written=462987
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=267
                HDFS: Number of bytes written=3
                HDFS: Number of read operations=9
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=2
                Launched reduce tasks=1
                Data-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=16830
                Total time spent by all reduces in occupied slots (ms)=23240
                Total time spent by all map tasks (ms)=8415
                Total time spent by all reduce tasks (ms)=5810
                Total vcore-milliseconds taken by all map tasks=8415
                Total vcore-milliseconds taken by all reduce tasks=5810
                Total megabyte-milliseconds taken by all map tasks=17233920
                Total megabyte-milliseconds taken by all reduce tasks=23797760
        Map-Reduce Framework
                Map input records=1
                Map output records=1
                Map output bytes=5
                Map output materialized bytes=19
                Input split bytes=264
                Combine input records=0
                Combine output records=0
                Reduce input groups=1
                Reduce shuffle bytes=19
                Reduce input records=1
                Reduce output records=1
                Spilled Records=2
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=343
                CPU time spent (ms)=4450
                Physical memory (bytes) snapshot=3202801664
                Virtual memory (bytes) snapshot=12837883904
                Total committed heap usage (bytes)=2916614144
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=3
        File Output Format Counters 
                Bytes Written=3
17/11/21 15:31:16 INFO mapred.MRBench: Running job 1: input=hdfs://ict-nn-01.cars.com:8020/benchmarks/MRBench/mr_input output=hdfs://ict-nn-01.cars.com:8020/benchmarks/MRBench/mr_output/output_-29456271
17/11/21 15:31:16 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:31:16 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:31:16 INFO client.RMProxy: Connecting to ResourceManager at ict-nn-01.cars.com/192.168.3.130:8050
17/11/21 15:31:16 INFO client.AHSProxy: Connecting to Application History server at ict-nn-01.cars.com/192.168.3.130:10200
17/11/21 15:31:17 INFO mapred.FileInputFormat: Total input paths to process : 1
17/11/21 15:31:17 INFO mapreduce.JobSubmitter: number of splits:2
17/11/21 15:31:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1511147390118_0007
17/11/21 15:31:17 INFO mapreduce.JobSubmitter: Kind: kms-dt, Service: 192.168.3.129:9292, Ident: (owner=admin, renewer=yarn, realUser=, issueDate=1511249440159, maxDate=1511854240159, sequenceNumber=7, masterKeyId=3)
17/11/21 15:31:17 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 192.168.3.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 179 for admin)
17/11/21 15:31:17 INFO impl.TimelineClientImpl: Timeline service address: http://ict-nn-01.cars.com:8188/ws/v1/timeline/
17/11/21 15:31:18 INFO impl.YarnClientImpl: Submitted application application_1511147390118_0007
17/11/21 15:31:18 INFO mapreduce.Job: The url to track the job: http://ict-nn-01.cars.com:8088/proxy/application_1511147390118_0007/
17/11/21 15:31:18 INFO mapreduce.Job: Running job: job_1511147390118_0007
17/11/21 15:31:41 INFO mapreduce.Job: Job job_1511147390118_0007 running in uber mode : false
17/11/21 15:31:41 INFO mapreduce.Job:  map 0% reduce 0%
17/11/21 15:31:47 INFO mapreduce.Job:  map 100% reduce 0%
17/11/21 15:31:56 INFO mapreduce.Job:  map 100% reduce 100%
17/11/21 15:31:58 INFO mapreduce.Job: Job job_1511147390118_0007 completed successfully
17/11/21 15:31:58 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=13
                FILE: Number of bytes written=462984
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=267
                HDFS: Number of bytes written=3
                HDFS: Number of read operations=9
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=2
                Launched reduce tasks=1
                Data-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=16978
                Total time spent by all reduces in occupied slots (ms)=23460
                Total time spent by all map tasks (ms)=8489
                Total time spent by all reduce tasks (ms)=5865
                Total vcore-milliseconds taken by all map tasks=8489
                Total vcore-milliseconds taken by all reduce tasks=5865
                Total megabyte-milliseconds taken by all map tasks=17385472
                Total megabyte-milliseconds taken by all reduce tasks=24023040
        Map-Reduce Framework
                Map input records=1
                Map output records=1
                Map output bytes=5
                Map output materialized bytes=19
                Input split bytes=264
                Combine input records=0
                Combine output records=0
                Reduce input groups=1
                Reduce shuffle bytes=19
                Reduce input records=1
                Reduce output records=1
                Spilled Records=2
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=271
                CPU time spent (ms)=2940
                Physical memory (bytes) snapshot=3204026368
                Virtual memory (bytes) snapshot=12845092864
                Total committed heap usage (bytes)=2918187008
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=3
        File Output Format Counters 
                Bytes Written=3
DataLines       Maps    Reduces AvgTime (milliseconds)
1               2       1       39012