hadoop集群的一些测试

hadoop的一些测试，做一个记录：

下操作都是在这个目录下面执行：

/opt/cloudera/parcels/CDH/jars

• 1.HDFS 写数据IO测试

Throughput 平均值： 86.73306
Average IO rate 平均值： 89.33704
IO rate std deviation 平均值： 16.62124
Test exec time 平均值： 56.8134

执行命令：

hadoop jar hadoop-test-2.6.0-mr1-cdh5.11.0.jar TestDFSIO -write -nrFiles 10 -fileSize 1000 -resFile /mydata/cdhtest/TestDFSIO_results_write.log


TestDFSIO  表示测试类型
-write     表示写测试
-nrFiles    表示往HDFS 写入多少个文件
-nrFiles    表示写入每个文件的大小
-resFile    表示最后的测试结果

原始测试报告：

----- TestDFSIO ----- : write
Date & time: Thu Jun 08 16:36:53 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 96.46178184203418
Average IO rate mb/sec: 98.18586730957031
IO rate std deviation: 14.038845095126302
Test exec time sec: 54.95

----- TestDFSIO ----- : write
Date & time: Thu Jun 08 16:39:51 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 75.68189386371205
Average IO rate mb/sec: 78.80995178222656
IO rate std deviation: 18.0744390712236
Test exec time sec: 60.497

----- TestDFSIO ----- : write
Date & time: Thu Jun 08 16:40:52 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 95.333428666762
Average IO rate mb/sec: 96.82975769042969
IO rate std deviation: 12.632113490531692
Test exec time sec: 53.577

----- TestDFSIO ----- : write
Date & time: Thu Jun 08 16:42:02 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 70.67537387272779
Average IO rate mb/sec: 74.50149536132812
IO rate std deviation: 19.976276004012956
Test exec time sec: 62.531

----- TestDFSIO ----- : write
Date & time: Thu Jun 08 16:42:59 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 95.51280826758868
Average IO rate mb/sec: 98.35811614990234
IO rate std deviation: 18.38454607319201
Test exec time sec: 52.512

• 2.HDFS 读数据IO测试

Throughput 平均值： 833.8921
Average IO rate 平均值： 862.3514
IO rate std deviation 平均值： 155.2752
Test exec time 平均值： 26.4224

测试命令：

hadoop jar hadoop-test-2.6.0-mr1-cdh5.11.0.jar TestDFSIO -read -nrFiles 10 -fileSize 1000 -resFile /mydata/cdhtest/TestDFSIO_results_read.log

TestDFSIO 表示测试类型
-write    表示读测试
-nrFiles  表示在HDFS 读多少个文件
-nrFiles  表示读的每个文件的大小
-resFile  表示最后的测试结果

原始测试报告：

----- TestDFSIO ----- : read
Date & time: Thu Jun 08 17:10:37 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 816.7932696234583
Average IO rate mb/sec: 857.7711791992188
IO rate std deviation: 185.61362190093095
Test exec time sec: 27.765

----- TestDFSIO ----- : read
Date & time: Thu Jun 08 17:11:08 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 811.4907084313885
Average IO rate mb/sec: 838.5701293945312
 IO rate std deviation: 153.72544710372162
Test exec time sec: 26.327

----- TestDFSIO ----- : read
Date & time: Thu Jun 08 17:11:38 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 813.2726089785295
Average IO rate mb/sec: 849.6246337890625
IO rate std deviation: 171.95324701441794
Test exec time sec: 26.305

----- TestDFSIO ----- : read
Date & time: Thu Jun 08 17:12:09 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 850.5571149102663
Average IO rate mb/sec: 869.06787109375
IO rate std deviation: 133.5895221661382
Test exec time sec: 26.382

----- TestDFSIO ----- : read
Date & time: Thu Jun 08 17:12:38 CST 2017
Number of files: 10
Total MBytes processed: 10000.0
Throughput mb/sec: 877.3469029654325
Average IO rate mb/sec: 896.7230224609375
IO rate std deviation: 131.4940910783484
Test exec time sec: 25.333

• 3.Namenode压力测试

测试命令：

hadoop jar hadoop-test-2.6.0-mr1-cdh5.11.0.jar nnbench  -operation create_write -maps 4 -reduces 2 -blockSize 1 -bytesToWrite 0 -numberOfFiles 100 -replicationFactorPerFile 3 -readFileAfterOpen true -baseDir /benchmarks/NNBench-`hostname -s`


-maps         map的个数
-reduces       reduce的个数
-numberOfFiles 创建的文件数

原始测试报告：

RAW DATA: TPS Total (ms): 8276
RAW DATA: Longest Map Time (ms): -1.496913630948E12
RAW DATA: Late maps 0
RAW DATA: # of exceptions: 4000

• 4.MapReduce程序测试

多次重复执行一个小作业，用于检查在机群上小作业的运行是否可重复以及运行是否高效

执行命令：

hadoop jar hadoop-test-2.6.0-mr1-cdh5.11.0.jar mrbench -numRuns 20

-numRuns 顺序运行一个小作业的次数

DataLines 1
Maps 2
Reduces 1
AvgTime (milliseconds)（从第一次到第五次） 17098、20921、16982、17538、18736

• 5.Mapreduce 排序测试

首先生成测试数据：

先生成测试数据1G 到 /examples/terasort-input 目录下

hadoop jar hadoop-examples-2.6.0-mr1-cdh5.11.0.jar teragen 10000000 /examples/terasort-input

执行排序任务：

hadoop jar hadoop-examples-2.6.0-mr1-cdh5.11.0.jar terasort /examples/terasort-input /examples/terasort-output
/examples/terasort-input  输入目录
/examples/terasort-output 输出目录

校验排序任务：

hadoop jar hadoop-examples-2.6.0-mr1-cdh5.11.0.jar teravalidate /examples/terasort-output /examples/terasort-validate
/examples/terasort-output     输入目录
/examples/terasort-validate   输出目录

最后查看校验数据：

/examples/terasort-validate/part-r-00000

发现校验数据为空，这排序功能正常。