Spark运行异常：java.lang.AbstractMethodError

最新推荐文章于 2023-08-05 11:10:32 发布

生命不息丶折腾不止

最新推荐文章于 2023-08-05 11:10:32 发布

阅读量5.8k

点赞数 1

分类专栏： spark 文章标签： spark 异常 exception

本文链接：https://blog.csdn.net/leen0304/article/details/78723536

版权

spark 专栏收录该内容

58 篇文章 10 订阅

订阅专栏

异常日志：

17/12/05 18:30:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.AbstractMethodError: JavaKryoSerializer$1.call(Ljava/lang/Object;)Ljava/lang/Iterable;
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)
	at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
	at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
17/12/05 18:30:44 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,ANY, 2249 bytes)
17/12/05 18:30:44 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main]
java.lang.AbstractMethodError: JavaKryoSerializer$1.call(Ljava/lang/Object;)Ljava/lang/Iterable;
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:129)
	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:284)
    at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
    at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:268)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
...

异常分析：

这个错误是因为你编译代码时用的Spark 1.*，但是运行时用的Spark 2.*。flatMap的函数签名在这两个版本中是不一样的。

// 在Spark1.*中new FlatMapFunction 重写call方法返回类型为：Iterable
JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
//
    @Override
    public Iterable<String> call(String line) throws Exception {
            return Arrays.asList(line.split("\\\t"));
    }
});

// 在Spark2.*中new FlatMapFunction 重写call方法返回类型为：Iterable
JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
//
    @Override
    public Iterator<String> call(String line) throws Exception {
            return Arrays.asList(line.split("\\\t")).iterator();
    }
});