ClouderaManager 启动spark时内存限制问题

最新推荐文章于 2022-06-07 10:19:12 发布

doegoo

最新推荐文章于 2022-06-07 10:19:12 发布

阅读量5k

点赞数

分类专栏： cloudera

本文链接：https://blog.csdn.net/doegoo/article/details/49181173

版权

cloudera 专栏收录该内容

9 篇文章 0 订阅

订阅专栏

Spark On Yarn执行中executor内存限制问题

15/10/16 16:56:04 INFO RMProxy: Connecting to ResourceManager at InterFinance01/10.238.18.80:8032

15/10/16 16:56:04 INFO Client: Requesting a new application from cluster with 3 NodeManagers
15/10/16 16:56:04 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (1024 MB per container)
java.lang.IllegalArgumentException: Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!
at org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:191)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)

解决方法如下：

从以上红色字体可以看到验证application需要的内存超过container的最大内存

所以配置yarn中的yarn.scheduler.maximum-allocation-mb这一个参数，使container的最大内存达到(1024+384 MB)错误提示的内存要求