使用 KafkaUtils.createDirectStream,hdfs 设置checkPoint目录,
restart spark streaming application; then error:
17/09/06 10:30:39 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:140)
at org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:827)
at org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:827)
at scala.Option.map(Option.scala:146)
查看application 日志,发现:Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher
错误同 http://blog.csdn.net/sunflower_cao/article/details/38046341 一样,
org.apache.spark.deploy.yarn.ExecutorLauncher 位于 jar包 spark-yarn_2.11-2.1.0.jar
spark-submit add jar, --jars /opt/spark/jars/spark-yarn_2.11-2.1.0.jar, 仍然报错。
问题目前未解决。
最终解决办法: 默认 --deploy-mode client 改为 --deploy-mode cluster