cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartiti

最新推荐文章于 2024-07-25 18:31:24 发布
最新推荐文章于 2024-07-25 18:31:24 发布
阅读量4.5k 收藏 3
点赞数
分类专栏： Intellij
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/appleyuchi/article/details/107833471
版权

碰到这么个问题:

20/08/06 10:26:45 WARN TaskSetManager: Lost task 1.0 in stage 1.0 (TID 2, 192.168.0.103, executor 1): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
    at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)
    at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2251)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

20/08/06 10:26:46 ERROR TaskSetManager: Task 1 in stage 1.0 failed 4 times; aborting job
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 5, 192.168.0.102, executor 0): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
    at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)
    at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2251)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2023)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1972)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1971)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2203)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2152)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2141)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2093)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2133)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2158)
    at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
    at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:361)
    at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:360)
    at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
    at javaread.main(javaread.java:37)
Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
    at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)
    at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2251)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

Process finished with exit code 1

 

原因是:

intellij中连接了远程集群，想要输出，必须先在intellij中mvn package。

然后再Ctrl+Shift+F10运行，这样就能顺利输出了。


 

spark报错：ClassCastException: cannot assign instance of SerializedLambda to field
Talk Is Cheap
10-20 1261
spark3
【FLink】cannot assign instance LinkedMap FlinkKafkaConsumerBase.pendingOffsetsToCommit
九师兄
06-03 638
1.场景1 1.1 概述 时间：20210603 一个老项目，打包后报错如下 Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.commons.collections.map.LinkedMap to field org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.pendingOffsetsToCommit of ty.
java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field
微电子学与固体电子学-俞驰
07-30 8536
在java连接集群的时候出错，完整报错如下: 20/07/30 11:04:11 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, 192.168.0.102, executor 0): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartit
Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.kafka.clients.consumer
最新发布
qq_37288262的博客
07-25 379
Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.kafka.clients.consumer.OffsetResetStrategy to field org.apache.flink.connector.kafka.source.enumerator.initializer.ReaderHandledOffsetsInitializer.offsetResetStrategy of type org
java.lang.ClassCastException: cannot assign instance of java.lang.Integer to of type java.math.BigD
weixin_41154794的博客
04-18 5936
java.lang.ClassCastException: cannot assign instance of java.lang.Integer to field com.quanquan.store.dto.PMSCouponDTO.discountAmount of type java.math.BigDecimal in instance of com.quanquan.store.dto...
Cause: cannot assign instance of java.lang.StackTraceElement to field java.lang.StackTraceElement.mo
柳鲲鹏
10-09 746
这是一个结果错误，是结合之前的信息看看，具体问题。
ireport报表错误-cannot assign instance of net.sf.jasperreports.engine.base.JRBaseSta
yzhp2z的博客
01-07 530
ireport报表错误cannot assign instance of net.sf.jasperreports.engine.base.JRBaseStaticText to field net.sf.jasperreports.engine.base.JRBaseParagraph.paragraphContainer of type net.sf.jasperreports.engine....
flink的实施报错cannot assign instance of org.apache.commons.collections.map.LinkedMap to field com.verver
吃素的哈士奇de博客
03-23 4323
flink的实施报错报错信息本质原因解决方式官方文档 报错信息 cannot assign instance of org.apache.commons.collections.map.LinkedMap to field com.ververica.cdc.debezium.DebeziumSourceFunction.pendingOffsetsToCommit of type org.apache.commons.collections.map.LinkedMap in instance of com
解决 Didn‘t find class “java.lang.invoke.SerializedLambda“
weixin_44012619的博客
07-07 839
升级android studio4.0之后发现 编译后，提升gradle版本到3.5.2，运行时发生crash。 android studio3.0已经支持了脱糖，为什么高版本会crash? 赶紧查看android studio 升级了什么功能导致的，附官方文档链接 官方文档 D8 和 R8 中的 Java 8 库脱糖 Android Studio 现在支持使用多种 Java 8 语言 API，而无需为应用设置最低 API 级别。 通过一个称为“脱糖”的过程，Android Studio 3.0 及更高版本
解决Didn't Find Class java.lang.invoke.SerializedLambda 问题
技术小黑屋
05-04 1602
问题表现 在低于 Android 7（Android Nougat）以下出现 错误的崩溃日志信息如下 1 2 3 4 5 6 7 8 9 Caused by: java.lang.ClassNotFoundException: Didn't find class "java.lang.invoke.SerializedLambda" on path: DexPathL...
tomcat java.math.BigDecimal cannot be cast to java.lang.Double问题解决
热门推荐
三水小鱼干的博客
06-12 1万+
出现这个错误的原因是将数据库中数值型取出保存到 map<String,Object>中，需要进行数值运算，转成double类型时抛出ava.math.BigDecimal cannot be cast to java.lang.Double。 解决办法； 1.转成string String num_str=map.get("key").toString(); 2.转成double，需承接上面的转成String Double num_double=Double.parseDouble(num_s
spark问题----cannot assign instance
u012450976的专栏
02-02 1817
User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 7.0 failed 4 times, most recent failure: Lost task 0.3 in stage 7.0 (TID 14, sq-spark-...
出现这个报错，无法解决cannot assign instance of org.apache.kafka.clients.consumer.0ffsetResetstrategy to field
weixin_70390501的博客
07-23 304
首先，将Kafka的kafka-clients从flink-connector-kafka排除，然后增加kafka-clients。第二：在flink的conf文件中增加parent-frist。第三：在flink的lib里面增加jar包。这是由于jar包冲突导致的。
解决办法java.lang.ClassCastException: cannot assign instance of net.sf.jasperreports
Mr.Cheney的专栏
09-08 3421
在使用ireport3.7编译生成jasper文件后放在S2SH的项目中使用，一切配置完成，出现了以下错误 java.lang.ClassCastException: cannot assign instance of net.sf.jasperreports.engine.base.JRBaseImage to field net.sf.jasperreports.engine.base.JR...
java.lang.ClassCastException:
bingdongguke的专栏
03-20 684
package com.mySurvey.dao.impl;import org.hibernate.SessionFactory;import org.springframework.orm.hibernate3.support.HibernateDaoSupport;import com.mySurvey.bean.User;import com.mySurvey.dao.UserDA
【JMeter】Error in rconfigure() method java.lang.ClassCastException: cannot assign instance of - 报错
Thread_v的博客
05-31 246
由于本机下载了两个jmeter版本，代理机解压了 5.6.3 两个 配置之后启动。然后由于 一直使用之前的5.5版本，没有注意改了配置文件之后启动，本机做分布式测试演练，只有一台机器，所以设置了不同的端口。又解压了同一个版本的jmeter 再次运行分布式运行成功。运行全部分布式机器之后就发生了报错；控制机 和 代理机 的版本不一致，
java spark-streaming java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRD
临地的博客
08-11 574
再次吐槽，spark的文档对初学者真的非常不友好。 同时下面的错误大概率也是可以解决掉的。 java.io.IOException: 远程主机强迫关闭了一个现有的连接 问题描述 在本地idea执行spark streaming相关代码时， setMaster为“local[2]”时，我是可以正常执行的，但是当我在 StandAlone 模式下启动本地集群的 setMaster 时，就会出现这个错误。 然后我分别根据不同的搜索结果，设置jars等，甚至看到官网关于此问题的两个issue ht
Spark：java.lang.ClassCastException
乔的博客
06-22 6684
Spark程序运行中可能会遇到这个异常： Caused by: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.co...
Cause: cannot assign instance of java.lang.StackTraceElement to field java.lang.StackTraceElement.moduleVersion of type java.lang.String in instance of java.lang.StackTraceElement
07-14
这个错误通常发生在尝试将一个`java.lang.StackTraceElement`对象中的`moduleVersion`字段赋值为`java.lang.StackTraceElement`类型的实例时。根据错误提示，`moduleVersion`字段的类型是`java.lang.String`，而不是`java.lang.StackTraceElement`。 要解决这个问题，你需要检查你的代码，确保在给`moduleVersion`字段赋值时使用了正确的类型。你可能需要查看`java.lang.StackTraceElement`类的文档或相关的代码，确认`moduleVersion`字段的正确使用方法。 如果你能提供更多的代码或上下文信息，我可以给出更具体的建议。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值