Container killed by YARN for exceeding memory limits

最新推荐文章于 2024-07-11 15:23:01 发布

Rone-X

最新推荐文章于 2024-07-11 15:23:01 发布

阅读量579

点赞数

分类专栏： Spark 文章标签： spark

本文链接：https://blog.csdn.net/RONE321/article/details/104820325

版权

Spark 专栏收录该内容

18 篇文章 0 订阅

订阅专栏

SparkOnYarn资源不足
本以为申请的资源够了呢，没想到任务跑了3个小时就死掉了
去Yarn上看日志
看到下面这个玩意儿

20/03/12 11:33:03 ERROR YarnClusterScheduler: Lost executor 2 on node-ana-corePKeW: Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
20/03/12 11:33:05 ERROR YarnClusterScheduler: Lost executor 3 on node-ana-coreQELy: Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
20/03/12 11:33:08 ERROR YarnClusterScheduler: Lost executor 1 on node-ana-coreqdOz: Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.