java.lang.IllegalStateException: unread block data

最新推荐文章于 2024-02-20 16:18:36 发布

wangzhanchi8

最新推荐文章于 2024-02-20 16:18:36 发布

阅读量9.6k

点赞数 1

分类专栏： Spark 文章标签： spark unread block data

本文链接：https://blog.csdn.net/wangzhanchi8/article/details/79617379

版权

Spark 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

Spark 通过textfile读取外部数据源，报错：unread block data，具体错误信息如下

18/03/19 20:46:54 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalStateException: unread block data
at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
18/03/19 20:46:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.IllegalStateException: unread block data
at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

问题原因分析：该问题查了好长时间(开始以为是数据源有问题、后来以为是本地开发环境有问题等等)；后来各种查、各种试，发现是缺少架包引起的，架包名称为：spark-assembly-1.6.0-hadoop2.6.0.jar、spark-examples-1.6.0-hadoop2.6.0.jar,将这两个架包加上后好了。

总结：引起该问题的原因有很多，最好由简入繁、从最简单直接的原因查起，逐一排除，提高排查问题效率

架包下载地址为：https://download.csdn.net/download/wangzhanchi8/10296218