sparkStreaming提交任务错误ck

最新推荐文章于 2022-02-23 17:32:53 发布
最新推荐文章于 2022-02-23 17:32:53 发布
阅读量381 收藏
点赞数
分类专栏： spark 文章标签： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_33881568/article/details/115417477
版权 
21/04/03 16:23:18 ERROR Utils: Exception encountered
java.io.IOException: unexpected exception type
	at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1736)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2078)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:561)
	at org.apache.spark.streaming.DStreamGraph.$anonfun$readObject$1(DStreamGraph.scala:199)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1343)
	at org.apache.spark.streaming.DStreamGraph.readObject(DStreamGraph.scala:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.streaming.Checkpoint$.$anonfun$deserialize$1(Checkpoint.scala:180)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.streaming.Checkpoint$.deserialize(Checkpoint.scala:184)
	at org.apache.spark.streaming.CheckpointReader$.$anonfun$read$2(Checkpoint.scala:363)
	at org.apache.spark.streaming.CheckpointReader$.$anonfun$read$2$adapted(Checkpoint.scala:359)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at org.apache.spark.streaming.CheckpointReader$.read(Checkpoint.scala:359)
	at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:840)
	at com.dongline.test.KafkaStreamTest$.main(KafkaStreamTest.scala:78)
	at com.dongline.test.KafkaStreamTest.main(KafkaStreamTest.scala)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
	... 59 more
Caused by: java.lang.IllegalArgumentException: Illegal lambda deserialization
	at scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:89)
	at scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:114)
	at scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38)
	at com.dongline.test.KafkaStreamTest$.$deserializeLambda$(KafkaStreamTest.scala)
	... 69 more
21/04/03 16:23:18 WARN CheckpointReader: Error reading checkpoint from file hdfs://192.168.2.31:9000/kafka/checkpoint/checkpoint-1617438170000
java.io.IOException: unexpected exception type
	at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1736)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2078)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:561)
	at org.apache.spark.streaming.DStreamGraph.$anonfun$readObject$1(DStreamGraph.scala:199)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1343)
	at org.apache.spark.streaming.DStreamGraph.readObject(DStreamGraph.scala:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.streaming.Checkpoint$.$anonfun$deserialize$1(Checkpoint.scala:180)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.streaming.Checkpoint$.deserialize(Checkpoint.scala:184)
	at org.apache.spark.streaming.CheckpointReader$.$anonfun$read$2(Checkpoint.scala:363)
	at org.apache.spark.streaming.CheckpointReader$.$anonfun$read$2$adapted(Checkpoint.scala:359)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at org.apache.spark.streaming.CheckpointReader$.read(Checkpoint.scala:359)
	at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:840)
	at com.dongline.test.KafkaStreamTest$.main(KafkaStreamTest.scala:78)
	at com.dongline.test.KafkaStreamTest.main(KafkaStreamTest.scala)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
	... 59 more
Caused by: java.lang.IllegalArgumentException: Illegal lambda deserialization
	at scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:89)
	at scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:114)
	at scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38)
	at com.dongline.test.KafkaStreamTest$.$deserializeLambda$(KafkaStreamTest.scala)
	... 69 more

出现的问题：sparkstreaming 修改了程序，但是仍然使用同一个checkpointdir，导致spark无法解析ck出现以上错误。
解决办法：程序中重新设置一个ckdir 或 删除原来的ckdir

在这里插入图片描述

qq_33881568
关注
专栏目录
spark-submit提交Spark Streamming+Kafka程序
主要分享大数据相关的知识，如Spark、Hudi
06-28 3264
转载请务必注明原创地址为：https://dongkelun.com/2018/06/19/sparkSubmitKafka/ 前言 Spark Streaming本身是没有Kafka相关的jar包和API的，如果想利用Spark Streaming获取Kafka里的数据，需要自己将依赖添加SBT或Maven项目中，添加依赖更新项目之后，就可以在Eclipse等IDE里直接运行Spark St...
flink实战--flink原理解析
阿华田的博客
09-30 2万+
Flink出现的背景 我们知道目前流处理的主要流行的计算引擎有，Storm，SparkStreaming。但是这个两个计算引擎都有自己的局限性。Storm实现了低延迟，但是目前还没有实现高吞吐，也不能在故障发生的时候准确的处理计算状态（将数据从一个事件保存到另一个事件的，这些保留下来的是数据较计算状态），同时也不能实现exactly-once。SparkStreaming通过微批处理方法............
Spark Streaming中序列化问题：org.apache.spark.SparkException: Task not serializable
见贤思齐
07-07 796
利用saprk streaming实时分析数据时报的一些问题：打印日志如下： org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:403) at org.apache.sp...
org.apache.spark.logging类报错
aoyugang0603的博客
08-12 1112
一，1 在使用spark读取kafka数据时，当spark升级到2.0之后，出现如上问题：之前遇到了，当时在工程里面添加了org.apache.spark.Logging类，能够运行。 但是在后期使用过程中，又遇到了相同的问题，为了一劳永逸，今天彻底把问题解决。 在项目下创建org.apache.spark.logging类 将源码考入类中 package org.apach...
Spark升级到2.0后测试stream-kafka测试报java.lang.NoClassDefFoundError: org/apache/spark/Logging错误
a280966503的博客
12-15 2430
在使用spark读取kafka数据时，当spark升级到2.0之后，出现如上问题：之前遇到了，当时在工程里面添加了org.apache.spark.Logging类，能够运行。 但是在后期使用过程中，又遇到了相同的问题，为了一劳永逸，今天彻底把问题解决。 开发环境：Idea intelligent14 + spark-2.1.0 + kafka 在Idea中安装scala
Spark Stream对接Kafka的两种版本选择
summer089089的博客
03-12 3146
对接kafka 0.8以及0.8以上的版本 Spark要在2.3.0一下选择较好，因为这个Spark对接kafka用这个比较稳定，1.0还是测试 导入依赖 <dependency> <groupId>org.apache.spark</groupId> <!--0.8是kafka的版本，2...
Spark知识点总结
CH_Axiaobai的博客
02-13 911
1.什么是spark？ spark是基于内存计算的通用大数据并行计算框架，是一个快速、通用可扩展的大数据分析引擎。它给出了大一统的软件开发栈，适用于不同场合的分布式场景，如批处理、迭代算法、交互式查询、流处理、机器学习和图计算。 2.Spark生态系统？ SparkCore：spark的核心计算 主要Rdd SparkSQL：提供了类sql方式操作结构化半结构化数据。对历史数据进行交互式查询。（即...
Spark 知识点干货
艾维的博客
10-24 981
spark ???? 翻译 Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools
大数据高频面试题-Spark
weixin_41540362的博客
07-09 1712
4.10Spark 4.10.1 Spark有几种部署方式？请分别简要论述 1）Local:运行在一台机器上，通常是练手或者测试环境。 2）Standalone:构建一个基于Mster+Slaves的资源调度集群，Spark任务提交给Master运行。是Spark自身的一个调度系统。 3）Yarn: Spark客户端直接连接Yarn，不需要额外构建Spark集群。有yarn-client和yarn-cluster两种模式，主要区别在于：Driver程序的运行节点。 4）Mesos：国内大环境比较.
执行spark程序报错: org.apache.spark.Logging
weixin_36815898的博客
03-10 2223
执行spark程序报错：Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging Spark Streaming在使用spark2自带的jar包进行wordcount统计时，出现以下报错 root@hadoop103 jars]# spark2-submit --master local[2] \ > --class org.apache.spark.examples.streaming.NetworkWordCount
Spark 处理log日志
giantleech的博客
05-18 2113
文章目录Spark 处理log日志(数据清洗)源数据准备数据清洗要求准备环境开始清洗实现需求 Spark 处理log日志(数据清洗) 源数据准备 准备一个 .log 格式的文件，内容如下所示 数据清洗要求 准备环境 读取文件后，按照格式一层层切分数据，并将最终的结果保存到 Mysql 数据库中，代码如下所示 先设置参数类，避免多次调用的代码冗余 object JdbcUtils { val url="jdbc:mysql://192.168.146.222:3306/etldemo" val
Scala避坑：Caused by: java.io.IOException: unexpected exception type
superY_26的博客
01-21 3017
问题描述： 本地利用sbt打包上传到服务器集群上运行，报错：Caused by: java.io.IOException: unexpected exception type。 本地环境Scala本版2.12.15，spark版本2.3.2。 本文的报错代码是： val data = df.where("key=val1").first() 解决方案： 在网上找了很多解决办法，替换函数等各种尝试，结果都没能解决问题。最后发现是因为Scala版本不一致导致的问题。 检查服务器上的Scala版本是2.11.8
java工具类系列 （三.ExceptionUtils）
学习、思考、分享
09-19 1万+
lang包提供的 异常处理工具类，需要该包下的几个工具类，源码如下： /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreem
Unexpected exception encountered during query.解决办法
bo_无问西东
08-04 1万+
办法： 更换最新mysql-connector-java-5.1.48jar包即可，就这个小问题搞得我两三个小时 还有个错误： 错误提示：Unknown initial character set index ‘255’ 在你的数据库名字后面加上这个东西即可 ?useUnicode=true&characterEncoding=utf8 ...
flink 使用问题记录
岁月静好，做自己。
02-23 2754
1. 提交flink任务： Process finished with exit code 130 org.apache.flink.runtime.rest.NotFoundException: Job 6fe06e87852fa7a12e2276e2876c067a not found at org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$1(AbstractE
SPARK常见报错
innersense的专栏
04-28 1253
在driver端报： [2020-04-28 10:55:16][main]-[ERROR][org.apache.spark.internal.Logging$class.logError(Logging.scala:91)] Error initializing SparkContext. org.apache.spark.SparkException: Yarn applicatio...
spark 使用中会遇到的一些问题及解决思路
热门推荐
xiao_jun_0820的专栏
04-14 6万+
7 内存溢出问题     在Spark中使用hql方法执行hive语句时，由于其在查询过程中调用的是Hive的获取元数据信息、SQL解析，并且使用Cglib等进行序列化反序列化，中间可能产生较多的class文件，导致JVM中的持久代使用较多，如果配置不当，可能引起类似于如下的OOM问题： Exception in thread "Thread-2" java.lang.
个人总结-Spark运行异常问题汇总
qq_34560135的博客
07-17 7083
Exception in thread “main” org.apache.spark.SparkException: When running with master ‘yarn’ either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. $ vim /etc/profile #添加以下两句话 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_
Intermec CK3手持终端配置指南
- LED状态指示包括新消息提示（湖泊色）、准备就绪（蓝色）、解码错误（红色）、电池电量低（红色）、充电中（红色/绿色）、未装电池（琥珀色）以及电池错误状态（琥珀色和红色交替闪烁）。 2. CK3输入法 - 提供...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值