测试hadoop集群的读写能力

本人安装的是cdh

本次实验是执行 10个文件每个文件是128mb 拿三台测试集群进行测试

执行如下

写能力测试

hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-client-jobclient-3.0.0-cdh6.1.0-tests.jar   TestDFSIO -write -nrFiles 10 -fileSize 128MB

得到的结果

20/05/26 16:22:23 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
20/05/26 16:22:23 INFO fs.TestDFSIO:             Date & time: Tue May 26 16:22:23 CST 2020
20/05/26 16:22:23 INFO fs.TestDFSIO:         Number of files: 10
20/05/26 16:22:23 INFO fs.TestDFSIO:  Total MBytes processed: 1280 //总文件大小
20/05/26 16:22:23 INFO fs.TestDFSIO:       Throughput mb/sec: 279.9  //吞吐量
20/05/26 16:22:23 INFO fs.TestDFSIO:  Average IO rate mb/sec: 313.91 //  平均IO速率
20/05/26 16:22:23 INFO fs.TestDFSIO:   IO rate std deviation: 102.48 //IO速率标准偏差
20/05/26 16:22:23 INFO fs.TestDFSIO:      Test exec time sec: 19.02 //整体时间

读能力测试

hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-mapreduce-client-jobclient-3.0.0-cdh6.1.0-tests.jar   TestDFSIO -read -nrFiles 10 -fileSize 128MB

结果

20/05/26 16:24:46 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
20/05/26 16:24:46 INFO fs.TestDFSIO:             Date & time: Tue May 26 16:24:46 CST 2020
20/05/26 16:24:46 INFO fs.TestDFSIO:         Number of files: 10
20/05/26 16:24:46 INFO fs.TestDFSIO:  Total MBytes processed: 1280
20/05/26 16:24:46 INFO fs.TestDFSIO:       Throughput mb/sec: 442.6
20/05/26 16:24:46 INFO fs.TestDFSIO:  Average IO rate mb/sec: 863.31
20/05/26 16:24:46 INFO fs.TestDFSIO:   IO rate std deviation: 661.44
20/05/26 16:24:46 INFO fs.TestDFSIO:      Test exec time sec: 18