Spark中任务报错java.io.IOException: Filesystem closed

问题来源

Spark程序操作HDFS的工具类中，手动获取了FileSystem，执行完HDFS相关操作后对FileSystem进行close，在后续的代码中报Filesystem closed错误。

ERROR LiveListenerBus: Listener EventLoggingListener threw an exception
java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:798)
	at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1985)
	at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1946)
	at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
	at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:140)
	at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:140)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:140)
	at org.apache.spark.scheduler.EventLoggingListener.onJobStart(EventLoggingListener.scala:163)
	at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:37)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
	at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1245)
	at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77)

解决方法

由于Spark程序是以Spark-On-Yarn模式运行，Spark会把job日志是写在HDFS上。他们使用的是同一个FileSystem，从而导致异常。将fs.close()去掉就会恢复正常。

关注公众号 数据工匠记 ，专注于大数据领域离线、实时技术干货定期分享！个人网站 www.lllpan.top