Exception in thread "Driver" java.io.IOException: Log directory hdfs://vanzin-krb-1.vpc.cloudera.com-CSDN博客

本文链接：https://blog.csdn.net/fhg12225/article/details/45191999

Exception in thread "Driver" java.io.IOException: Log directory hdfs://vanzin-krb-1.vpc.cloudera.com:8020/user/spark/applicationHistory/application_1417554558066_0003 already exists!
        at org.apache.spark.util.FileLogger.createLogDir(FileLogger.scala:129)
        at org.apache.spark.util.FileLogger.start(FileLogger.scala:115)
        at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:74)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:353)

设置：然后重试任务

sparkConf.set("spark.eventLog.overwrite",true);

Spark History Server support multiple application attempts

Spark on Yarn supports running multiple application attempts (configurable number) in case the first (or second..) attempts fail. The Spark History server only supports one history file though. Under the default configs it keeps the first attempts history file. You can set the undocumented config spark.eventLog.overwrite to allow the follow on attempts to overwrite the first attempts history file.

Note that in spark 1.2 not having the overwrite config set causes any following attempts to actually fail to run, in spark 1.3 they run and you just see a warning at the end of the attempts.

It would be really nice to have an option that keeps all the attempts history files. This way a user can go back and look at each one individually.

spark.eventLog.enabled	false	是否记录Spark事件，用于应用程序在完成后重构webUI。
spark.eventLog.compress	false	是否压缩记录Spark事件，前提spark.eventLog.enabled为true。
spark.eventLog.dir	file:///tmp/spark-events	如果spark.eventLog.enabled为 true，该属性为记录spark事件的根目录。在此根目录中，Spark为每个应用程序创建分目录，并将应用程序的事件记录到在此目录中。用户可以将此属性设置为HDFS目录，以便history server读取历史记录文件。