hadoop 基准测试与读写测试

最新推荐文章于 2022-06-09 10:15:38 发布

weixin_33859665

最新推荐文章于 2022-06-09 10:15:38 发布

阅读量98

点赞数

文章标签：大数据 python

原文链接：https://my.oschina.net/forrest420/blog/477967

版权

2019独角兽企业重金招聘Python工程师标准>>>

hadoop 基准测试与读写测试

排序100G数据

/opt/cloudera/parcels/CDH/bin/yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar teragen 1000000000 /tmp/test/terasort/100G-input

/opt/cloudera/parcels/CDH/bin/yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar terasort /tmp/test/terasort/100G-input /tmp/test/terasort/100G-output

/opt/cloudera/parcels/CDH/bin/yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar teravalidate /tmp/test/terasort/100G-output /tmp/test/terasort/100G-validate

测试读写性能

/opt/cloudera/parcels/CDH/bin/yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 10000

/opt/cloudera/parcels/CDH/bin/yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 10000

/opt/cloudera/parcels/CDH/bin/yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -clean

参考：

http://www.opstool.com/article/249

http://jeoygin.org/2012/12/hadoop-benchmarks.html

转载于:https://my.oschina.net/forrest420/blog/477967