SPARK常见报错

最新推荐文章于 2023-02-01 23:14:47 发布

innersense

最新推荐文章于 2023-02-01 23:14:47 发布

阅读量1.2k

点赞数

本文链接：https://blog.csdn.net/innersense/article/details/105808973

版权

博客内容提到了在使用Spark进行Yarn应用运行时遇到的错误，包括driver端的SparkException和Yarn application已结束的问题。错误源于Yarn客户端调度器无法启动，可能由于应用已被杀死或无法启动应用主。日志中还显示了ExitCodeException，指出HDFS上依赖的JAR包缺失。解决办法是检查并确保`spark.yarn.jars`配置指向的HDFS路径包含了所有必要的JAR包。

摘要由CSDN通过智能技术生成

在driver端报：

[2020-04-28 10:55:16][main]-[ERROR][org.apache.spark.internal.Logging$class.logError(Logging.scala:91)] Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:149)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:836)
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84)
at com.tencent.csi