spark在yarn上运行报错:Yarn application has already ended

最新推荐文章于 2020-02-12 16:34:59 发布

李功林

最新推荐文章于 2020-02-12 16:34:59 发布

阅读量6k

点赞数 3

分类专栏： HPE大数据学习零基础学习大数据文章标签： yarn 错误 spark在yarn上运行

本文链接：https://blog.csdn.net/weixin_43363946/article/details/83625471

版权

当尝试在YARN上运行Spark应用时遇到'Yarn application has already ended!'异常。经过排查，可能是由于JDK版本与Spark或YARN存在兼容性问题。解决方案是在yarn-site.xml中添加特定配置项，如yarn.nodemanager.pmem-check-enabled和yarn.nodemanager.vmem-check-enabled，调整内存检查设置。虽然JDK版本影响的具体原因尚不清楚，但此方法成功解决了问题。

摘要由CSDN通过智能技术生成

spark在yarn上运行报错:

Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
   at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)
   at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)
   at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
   at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
   at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29)
   at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAc