**org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 2) had a not serializable result:
- object not serializable (class: org.apache.hadoop.hbase.io.ImmutableBytesWritable, value: 7a 68 61 6e 67 73 61 6e)
- field (class: scala.Tuple2, name: _1, type: class java.lang.Object)
- object (class scala.Tuple2, (7a 68 61 6e 67 73 61 6e,keyvalues={lisi/course:chinese/1434685460521/Put/vlen=2/mvcc=0, lisi/course:english/1434698883293/Put/vlen=2/mvcc=0, lisi/course:math/1434685470168/Put/vlen=2/mvcc=0}))
- element of array (index: 0)
- array (class [Lscala.Tuple2;, size 2)
at org.apache.spark.scheduler.DAGScheduler.org a p a c h e apache apachespark s c h e d u l e r scheduler schedulerDAGScheduler f a i l J o b A n d I n d e p e n d e n t S t a g e s ( D A G S c h e d u l e r . s c a l a : 1204 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r failJobAndIndependentStages(DAGScheduler.scala:1204) at org.apache.spark.scheduler.DAGScheduler failJobAndIndependentStages(DAGScheduler.scala:1204)atorg.apache.spark.scheduler.DAGScheduleranonfun$abortStage 1. a p p l y ( D A G S c h e d u l e r . s c a l a : 1193 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r 1.apply(DAGScheduler.scala:1193) at org.apache.spark.scheduler.DAGScheduler 1.apply(DAGScheduler.scala:1193)atorg.apache.spark.scheduler.DAGScheduler a n o n f u n anonfun anonfunabortStage 1. a p p l y ( D A G S c h e d u l e r . s c a l a : 1192 ) a t s c a l a . c o l l e c t i o n . m u t a b l e . R e s i z a b l e A r r a y 1.apply(DAGScheduler.scala:1192) at scala.collection.mutable.ResizableArray 1.apply(DAGScheduler.scala:1192)atscala.collection.mutable.ResizableArrayclass.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)
at org.apache.spark.scheduler.DAGSchedulerKaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲handleTaskSetFa…anonfun$handleTaskSetFailed 1. a p p l y ( D A G S c h e d u l e r . s c a l a : 693 ) a t s c a l a . O p t i o n . f o r e a c h ( O p t i o n . s c a l a : 236 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r . h a n d l e T a s k S e t F a i l e d ( D A G S c h e d u l e r . s c a l a : 693 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . o n R e c e i v e ( D A G S c h e d u l e r . s c a l a : 1393 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r E v e n t P r o c e s s L o o p . o n R e c e i v e ( D A G S c h e d u l e r . s c a l a : 1354 ) a t o r g . a p a c h e . s p a r k . u t i l . E v e n t L o o p 1.apply(DAGScheduler.scala:693) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354) at org.apache.spark.util.EventLoop 1.apply(DAGScheduler.scala:693)atscala.Option.foreach(Option.scala:236)atorg.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)atorg.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)atorg.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)atorg.apache.spark.util.EventLoop$anon$1.run(EventLoop.scala:48)**
解决方法，在SparkConf上set序列化：
val sconf = new SparkConf()
.setMaster(“spark://h230:7077”)
.setAppName(“SparkFromHbase”)
.set(“spark.executor.memory”, “1g”)
.set(“spark.serializer”, “org.apache.spark.serializer.KryoSerializer”)
376
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
