问题 spark zeepelin0.8.1无法使用map等算子

最新推荐文章于 2020-07-22 19:14:14 发布
最新推荐文章于 2020-07-22 19:14:14 发布
阅读量206 收藏
点赞数
分类专栏： 大数据—Spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_24393347/article/details/103046023
版权
错误信息：

目前我们有两个版本的CDH集群，CDH5.9.1 ，CDH6.1.1
在安装了相同版本的zeepelin0.8.1 ，相同的代码，在CDH5.9.1中出现类找不到的错误

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, hdh09, executor 13): java.lang.ClassNotFoundException: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
	at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:88)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1433)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1421)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1420)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1420)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1644)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1603)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1592)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1844)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1857)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1870)
	at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1328)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.take(RDD.scala:1302)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:39)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:54)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:56)
	at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:58)
	at $iwC$$iwC$$iwC$$iwC.<init>(<console>:60)
	at $iwC$$iwC$$iwC.<init>(<console>:62)
	at $iwC$$iwC.<init>(<console>:64)
	at $iwC.<init>(<console>:66)
	at <init>(<console>:68)
	at .<init>(<console>:72)
	at .<clinit>(<console>)
	at .<init>(<console>:7)
	at .<clinit>(<console>)
	at $print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)
	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)
	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)
	at org.apache.zeppelin.spark.SparkScala210Interpreter.scalaInterpret(SparkScala210Interpreter.scala:94)
	at org.apache.zeppelin.spark.BaseSparkScalaInterpreter$$anonfun$_interpret$1$1.apply(BaseSparkScalaInterpreter.scala:96)
	at org.apache.zeppelin.spark.BaseSparkScalaInterpreter$$anonfun$_interpret$1$1.apply(BaseSparkScalaInterpreter.scala:90)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at scala.Console$.withOut(Console.scala:126)
	at org.apache.zeppelin.spark.BaseSparkScalaInterpreter._interpret$1(BaseSparkScalaInterpreter.scala:90)
	at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.interpret(BaseSparkScalaInterpreter.scala:121)
	at org.apache.zeppelin.spark.NewSparkInterpreter.interpret(NewSparkInterpreter.java:147)
	at org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:73)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:103)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:632)
	at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
	at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
	at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:88)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229)
	... 3 more
原因

i w C iwC iwC i w C iwC iwC i w C iwC iwC i w C iwC iwC i w C iwC iwC i w C iwC iwC i w C iwC iwC i w C iwC iwC$anonfun$1 在scala中是闭包被编译生成的类名，zeepelin0.8.1 在spark interpreter中默认会启用新的spark 2的解析器，而在CDH5.9.1中使用的spark是1.6.0版本，由于版本的不同，对闭包编译的差异导致了上面的问题

解决办法

修改 spark interpreter 配置
zeppelin.spark.useNew 改为false

章锡平
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark java.utils.List 没有map方法的问题
lv_tianxiaomiao的博客
05-10 445
spark使用scala来编码的时候，有时候会使用java中的数据结构 在使用java中的list的时候，使用map方法会提示出错 这个问题倒是很简单，java的list确实没有map方法，不过转换成scala的list就可以解决了 scala中的list和java中的list是可以直接隐式转换的，不用在代码里面体现 只需要引入相应的包就可以
spark的java的map函数_当我在Spark上使用DataFrame的map函数时发生java.lang.ClassNotFoundException...
weixin_39568172的博客
03-02 240
我有一个DataFrame＆＃34; orderedDf＆＃34; ，哪个架构如下：root|-- schoolID: string (nullable = true)|-- count(studentID): long (nullable = false)|-- count(teacherID): long (nullable = false)|-- sum(size): long (nulla...
spark中dataset对象使用map算子报错问题
weixin_43995212的博客
12-01 1447
报错： Error:(29, 32) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for s...
Spark解决Map算子中调用自定义方法出现的初始化问题
weixin_44736028的博客
05-25 1113
有一批JSON数据需要被要存成parquet格式，但是这些json数据中有一些数据是脏数据, 例如: {‘name’:‘zhangsan’,‘age’:14,‘wo}k’:‘teacher’} 如果存在这样的数据的话，当我直接用下面spark代码将json文件转成parquet文件时则会报错 代码: spark.read.json(“data/data.json”).write.mode(SaveMode.Overwrite).parquet(“data/data.parquet”) 异常: 19/08/2
关于在本地idea当中提交spark代码到远程的错误总结（第二篇）
weixin_30483697的博客
12-26 300
当代码能正常提交到spark集群运行的时候，出现下面的错误: Exception in thread "main" java.lang.OutOfMemoryError: PermGen space at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineCla...
spark-0.8.1-incubating.tgz
02-03
在实际使用中，Spark-0.8.1-incubating可能会与Hadoop生态系统中的其他组件（如HDFS、YARN或Mesos）集成，以实现更全面的大数据处理解决方案。开发者可以利用这个版本的Spark来构建分布式应用程序，处理PB级别的数据...
graphframes-0.8.1-spark3.0-s_2.12.jar
06-03
graphframes-0.8.1-spark3.0-s_2.12.jar
spark-streaming_2.9.3-0.8.1-incubating.jar
05-31
Spark 项目流 org.apache.spark/spark-streaming_2.9.3/0.8.1-incubating/spark-streaming_2.9.3-0.8.1-incubating.jar
spring-ai-core 0.8.1
05-21
《Spring AI Core 0.8.1：开启人工智能之旅》 在现代软件开发领域，Spring框架以其强大的功能和灵活性，已经成为Java开发中的首选框架之一。而Spring AI Core则是Spring生态系统中专门为人工智能（AI）和机器学习...
spring-ai-core-0.8.1
最新发布
05-31
spring-ai-core 0.8.1,解决大家使用2023.0.1.0 版本 Spring Cloud Alibaba 依赖，代码依赖下载报错问题， <groupId>com.alibaba.cloud</groupId> <artifactId>spring-cloud-alibaba-dependencies <version>...
spark shell 启动 出现org.apache.spark.SparkException: Found both spark.executor.extraClassPath and SPARK
liuhehe123的博客
02-26 3741
启动spark shell报错： Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 19/02/26 16:56:07 WARN util.NativeCodeLoader: Unable...
spark中读取hdfs中文件出错
smallboy2011的专栏
07-19 3594
scala> val rdd1 = sc.textFile("hdfs://master:9001/spark/spark02/directory/") 14/07/19 17:09:36 INFO MemoryStore: ensureFreeSpace(138763) called with curMem=0, maxMem=309225062 14/07/19 17:09:36 INFO
zeppelin+spark遇到的问题
MESSD10S的博客
12-10 1473
       由于电脑重装系统（Ubuntu18.04），之前运行得好好的zeppelin突然各种报错。由于本人比较菜，而且网上类似bug解决的帖子很少，所以最后花了好几个小时才搞定。希望能够帮助到出现相同问题的同学。        重装完系统后，我下了spark-2.4.0版本，zeppelin-0.8.0版本。（spark的安装配置可以参考这篇文章：https://blog.csdn.net...
部署zeppelin时遇到的spark on yarn的submit方式问题
Tony_老七
07-18 4832
部署zeppelin时候遇到的一个跟spark submit application模式相关的问题
Zeppelin结合Spark等各种Interpreter的使用
热门推荐
小丑鱼1127的博客
09-05 2万+
Zeppelin结合Spark等各种Interpreter的使用 Apache Zeppelin是基于Web的笔记本，支持SQL、Scala等数据驱动的交互式数据分析和协作文档。技术方面主要有Spark、SQL、Python。在部署方面支持单个用户也支持多用户。 Zeppelin Notebook可以满足数据摄取、数据发现、数据分析、数据可视化与协作。 多语言后端 Apace Zeppel...
Spark中的map相关的算子以及区别
liam12138的博客
05-21 838
Spark中map相关算子及区别 map算子：map[U](f: (T) => U): RDD[U] 是对RDD中的每个元素都执行一个指定的函数开产生一个新的RDD。 示例： val conf = new SparkConf().setAppName("WC").setMaster("local[*]") val sc = new SparkContext(conf) val lineRDD = sc.makeRDD(1 to 10) .map(data => {
Spark 1.6 Thrift Server 加载类异常处理
半日闲的博客
06-21 1205
错误日志分析 Spark 1.6 在启动Spark Thrift Server的时候，Beeline不断出现报错： 0: jdbc:hive2://10.59.34.204:10000> show tables; Error: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unabl...
zeppelin使用中的问题汇总
简单就好
03-03 5258
1、使用zeppelin 0.6.2的spark interpreter报错报错：java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.rdd.RDDOperationScope$ Interpreter output:com.fasterxml.jackson.databind.JsonMapping
spark开发中遇到的一些问题及解决方法总结
weixin_42094092的博客
07-22 1232
1.Exception in thread “main” java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet; 解决：scala版本不对，将scala 2.11 换成scala 2.10 2.win7 运行saprk程序会报如下报错误： Failed to locate the winutils binary in the hadoop b
怎么使用清华源pip安装torch1.7.1 和torchvision 0.8.1
04-02
1. 首先安装清华源 使用以下命令添加清华源： ``` ...``` 2. 安装torch1.7.1 ...使用以下命令安装torchvision 0.8.1： ``` pip install torchvision==0.8.1 -i https://pypi.tuna.tsinghua.edu.cn/simple ```
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值