Adam学习10之集群节点的内存对运行时间的影响

最新推荐文章于 2024-06-27 01:12:31 发布

KeepLearningBigData

最新推荐文章于 2024-06-27 01:12:31 发布

阅读量846

点赞数

分类专栏： adam 文章标签： Adam学习10之集群节点的内存对运行时

本文链接：https://blog.csdn.net/xubo245/article/details/51173267

版权

adam 专栏收录该内容

25 篇文章 0 订阅

订阅专栏

问题：将单节点的内存从512M加到4096M时运行时间不减少反而增加，不解，待解决。是不是单个测试用例的偶然性？

hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:26936ms
Method 2=> Length:2971 sum2:7888989 time:2367ms
Method 3=> Length:2971 sum3:7888989.0 time:4200ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ cp cluster.sh submit.sh
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ mv submit.sh cluster4G.sh 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ vi cluster4G.sh 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster4G.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:34320ms
Method 2=> Length:2971 sum2:7888989 time:3078ms
Method 3=> Length:2971 sum3:7888989.0 time:4136ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ 
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster4G.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:27065ms
Method 2=> Length:2971 sum2:7888989 time:2764ms
Method 3=> Length:2971 sum3:7888989.0 time:1292ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster
-bash: ./cluster: No such file or directory
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:24095ms
Method 2=> Length:2971 sum2:7888989 time:4263ms
Method 3=> Length:2971 sum3:7888989.0 time:4006ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$ ./cluster4G.sh 
fq0.count:105887                                                                
Method 1=> Length:2971 sum:7888989 time:26671ms
Method 2=> Length:2971 sum2:7888989 time:3404ms
Method 3=> Length:2971 sum3:7888989.0 time:4196ms
hadoop@Master:~/cloud/testByXubo/spark/hs38DH/package$

集群：

Application ID Name Cores Memory per Node Submitted Time User State Duration 
app-20160417144715-0006  readFileFromHs38DH  12  4.0 GB  2016/04/17 14:47:15 hadoop FINISHED 47 s 
app-20160417144235-0005  readFileFromHs38DH  12  512.0 MB  2016/04/17 14:42:35 hadoop FINISHED 40 s 
app-20160417143730-0004  readFileFromHs38DH  12  4.0 GB  2016/04/17 14:37:30 hadoop FINISHED 44 s 
app-20160417143619-0003  readFileFromHs38DH  12  4.0 GB  2016/04/17 14:36:19 hadoop FINISHED 51 s 
app-20160417143459-0002  readFileFromHs38DH  12  512.0 MB  2016/04/17 14:34:59 hadoop FINISHED 42 s