spark问题----cannot assign instance

天明cc

于 2020-02-02 10:15:27 发布

阅读量1.8k

点赞数

分类专栏：大数据文章标签： spark

本文链接：https://blog.csdn.net/u012450976/article/details/104141802

版权

在使用Spark 2.3和Scala 2.11时，遇到一个涉及UDF的奇怪问题。该UDF返回boolean类型，在本地模式和Scala代码中运行正常，但在Java版本中仅在集群上出现问题。将返回类型改为int后，问题解决。但原因未知。

摘要由CSDN通过智能技术生成

User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage 7.0 (TID 14, sq-spark-node13.800best.com, executor 2): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD

at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)

at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2006)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

at java.io.ObjectInputStream.readObject0(ObjectInputS