Spark集成Elasticsearch,在local下正常运行,提交到集群运行jar出现异常:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 10, 172.16.14.32): java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.threadpool.ThreadPool
经定位,发现是由于spark集群运行时默认加载$spark_home/jars下的guava-14.0.1.jar版本的包,而Elasticsearch使用的是guava-18.0.jar版本,提交到集群运行时,guava-14.0.1.jar版本覆盖掉了Elasticsearch中的guava-18.0.jar版本,造成java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.threadpool.ThreadPool 异常。
解决方式参考:https://blog.csdn.net/u010454030/article/details/70054822
博主所写的资料
1