Hadoop集群之吞吐能力测试

Hadoop集群之吞吐能力测试

一、测试HDFS写性能

测试内容：向HDFS集群写10个128M的文件

[hadoop@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
client-jobclient-3.1.3-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB


2020-04-16 13:41:24,724 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
2020-04-16 13:41:24,724 INFO fs.TestDFSIO:             Date & time: Thu Apr 16 13:41:24 CST 2020
2020-04-16 13:41:24,724 INFO fs.TestDFSIO:         Number of files: 10
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:  Total MBytes processed: 1280
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:       Throughput mb/sec: 8.88
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:  Average IO rate mb/sec: 8.96
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:   IO rate std deviation: 0.87
2020-04-16 13:41:24,725 INFO fs.TestDFSIO:      Test exec time sec: 67.61

注意：
nrFiles n为生成mapTask的数量，生产环境一般可通过8088端口查看cpu核数，设置为cpu核数-1

Number of files：生成mapTask数量，一般是集群中CPU核数-1，我们测试虚拟机就按照实际的物理cpu-1分配即可

Total MBytes processed：单个map处理的文件大小

Throughput mb/sec:单个mapTak的吞吐量    
	计算方式：处理的总文件大小/每一个mapTask写数据的时间累加
	集群整体吞吐量：生成mapTask数量*单个mapTak的吞吐量

Average IO rate mb/sec::单个mapTak的吞吐量  
	计算方式：每个mapTask处理文件大小/每一个mapTask写数据的时间 累加/生成mapTask数量

IO rate std deviation:方差、反映各个mapTask处理的差值，越小越均衡

二、测试HDFS读性能

测试内容：读取HDFS集群10个128M的文件

[hadoop@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
client-jobclient-3.1.3-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB


2020-04-16 13:43:38,857 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
2020-04-16 13:43:38,858 INFO fs.TestDFSIO:   Date & time: Thu Apr 16 13:43:38 CST 2020
2020-04-16 13:43:38,859 INFO fs.TestDFSIO:         Number of files: 10
2020-04-16 13:43:38,859 INFO fs.TestDFSIO:  Total MBytes processed: 1280
2020-04-16 13:43:38,859 INFO fs.TestDFSIO:       Throughput mb/sec: 85.54
2020-04-16 13:43:38,860 INFO fs.TestDFSIO:  Average IO rate mb/sec: 100.21
2020-04-16 13:43:38,860 INFO fs.TestDFSIO:   IO rate std deviation: 44.37
2020-04-16 13:43:38,860 INFO fs.TestDFSIO:      Test exec time sec: 53.61

三、删除测试生成数据

[hadoop@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
client-jobclient-3.1.3-tests.jar TestDFSIO -clean

四、使用Sort程序评测MapReduce

（1）使用RandomWriter来产生随机数，每个节点运行10个Map任务，每个Map产生大约1G大小的二进制随机数

[hadoop@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
examples-3.1.3.jar randomwriter random-data

（2）执行Sort程序

[hadoop@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
examples-3.1.3.jar sort random-data sorted-data

（3）验证数据是否真正排好序了

[hadoop@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-
client-jobclient-3.1.3-tests.jar testmapredsort -sortInput random-data -sortOutput sorted-data