WARN FileInputDStream:87 - Error finding new filesjava.lang.NullPointerException报错为什么

原创已于 2022-04-05 12:14:42 修改 · 542 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#java #spark

于 2022-04-05 11:31:40 首次发布

这篇文章探讨了2022年4月5日遇到的Spark Streaming中FileInputDStream读取文件错误，特别提到NullPointerException，着重于如何识别问题根源并解决，即文件夹操作而非单个文件。

2022-04-05 11:29:30 WARN FileInputDStream:87 - Error finding new files
java.lang.NullPointerException
   at scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:192)
   at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:192)
   at scala.collection.SeqLike$class.size(SeqLike.scala:106)
   at scala.collection.mutable.ArrayOps$ofRef.size(ArrayOps.scala:186)
   at scala.collection.mutable.Builder$class.sizeHint(Builder.scala:69)
   at scala.collection.mutable.ArrayBuilder.sizeHint(ArrayBuilder.scala:22)
   at scala.collection.TraversableLike$class.builder$1(TraversableLike.scala:230)
   at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
   at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
   at org.apache.spark.streaming.dstream.FileInputDStream.findNewFiles(FileInputDStream.scala:206)
   at org.apache.spark.streaming.dstream.FileInputDStream.compute(FileInputDStream.scala:149)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1$$anonfun$apply$7.apply(DStream.scala:342)
   at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:341)
   at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:336)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:334)
   at scala.Option.orElse(Option.scala:289)