Spark Exception in thread main java.lang.AbstractMethodError

最新推荐文章于 2023-03-01 13:51:59 发布

彩虹豆

最新推荐文章于 2023-03-01 13:51:59 发布

阅读量381

点赞数

分类专栏： spark 文章标签： spark 大数据

本文链接：https://blog.csdn.net/qq_41875667/article/details/106385138

版权

spark 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

都是CDH集群模式

测试Spark安装包：

SPARK2_ON_YARN-2.1.0.cloudera1.jar

SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.232957-el6.parcel

SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.232957-el6.parcel.sha

manifest.json

生产Spark安装包：

SPARK2_ON_YARN-2.1.0.cloudera2.jar

SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el6.parcel

SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el6.parcel.sha

manifest.json

区别在于测试 cloudera1,生产 clouder2.其他CDH组件版本都是一样的。

spark程序打包上传到测试集群三种模式运行都OK,然后打包运行在生产环境：使用 --master local 模式运行生产没问题，但是指定为client或cluster模式都会报错：

20/05/27 16:20:15 INFO yarn.Client: Preparing resources for our AM container
Exception in thread "main" java.lang.AbstractMethodError
	at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:81)
	at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:80)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager.obtainCredentials(ConfigurableCredentialManager.scala:80)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:408)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:856)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:172)
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1133)
	at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1192)
	at org.apache.spark.deploy.yarn.Client.main(Client.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

代码，以及配置文件方面都没有任何差错，经过仔细排查才发现，spark安装版本不一致，然后验证，在测试集群把生产的spark安装包拉过来，把之前的spark卸载，安装cloudera2，然后程序执行，和生产结果一致。只可以local模式运行，client和cluster模式都不行。问题确定，把生产的spark版本换成 cloudera1。