hadoop安装完成后基准测试

最新推荐文章于 2022-11-09 08:27:16 发布

AA赵师傅

最新推荐文章于 2022-11-09 08:27:16 发布

阅读量211

点赞数

分类专栏： hadoop运维文章标签： hadoop 基准测试压测

本文链接：https://blog.csdn.net/zcb_data/article/details/113071913

版权

hadoop运维专栏收录该内容

15 篇文章 1 订阅

订阅专栏

1）测试HDFS写性能

测试内容：向HDFS集群写10个128M的文件

[mars@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB



2020-04-16 13:41:24,724 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write

2020-04-16 13:41:24,724 INFO fs.TestDFSIO:             Date & time: Thu Apr 16 13:41:24 CST 2020

2020-04-16 13:41:24,724 INFO fs.TestDFSIO:         Number of files: 10

2020-04-16 13:41:24,725 INFO fs.TestDFSIO:  Total MBytes processed: 1280

2020-04-16 13:41:24,725 INFO fs.TestDFSIO:       Throughput mb/sec: 8.88

2020-04-16 13:41:24,725 INFO fs.TestDFSIO:  Average IO rate mb/sec: 8.96

2020-04-16 13:41:24,725 INFO fs.TestDFSIO:   IO rate std deviation: 0.87

2020-04-16 13:41:24,725 INFO fs.TestDFSIO:      Test exec time sec: 67.61

2）测试HDFS读性能

测试内容：读取HDFS集群10个128M的文件

[mars@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB



2020-04-16 13:43:38,857 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read

2020-04-16 13:43:38,858 INFO fs.TestDFSIO:   Date & time: Thu Apr 16 13:43:38 CST 2020

2020-04-16 13:43:38,859 INFO fs.TestDFSIO:         Number of files: 10

2020-04-16 13:43:38,859 INFO fs.TestDFSIO:  Total MBytes processed: 1280

2020-04-16 13:43:38,859 INFO fs.TestDFSIO:       Throughput mb/sec: 85.54

2020-04-16 13:43:38,860 INFO fs.TestDFSIO:  Average IO rate mb/sec: 100.21

2020-04-16 13:43:38,860 INFO fs.TestDFSIO:   IO rate std deviation: 44.37

2020-04-16 13:43:38,860 INFO fs.TestDFSIO:      Test exec time sec: 53.61

3）删除测试生成数据

[mars@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar TestDFSIO -clean

4）使用Sort程序评测MapReduce

（1）使用RandomWriter来产生随机数，每个节点运行10个Map任务，每个Map产生大约1G大小的二进制随机数

[mars@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar randomwriter random-data

（2）执行Sort程序

[mars@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar sort random-data sorted-data

（3）验证数据是否真正排好序了

[mars@hadoop102 mapreduce]$

hadoop jar /opt/module/hadoop-3.1.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar testmapredsort -sortInput random-data -sortOutput sorted-data