oozie on spark2:java.lang.IllegalArgumentException: Attempt to add jar multiple times to the distrib

错误信息详情

2018-08-20 19:55:03,088 [uber-SubtaskRunner] ERROR org.apache.spark.SparkContext  - Error initializing SparkContext.
java.lang.IllegalArgumentException: Attempt to add (hdfs://hdh116:8020/user/root/share/lib/lib_20180820162943/spark2/scala-compiler-2.11.8.jar) multiple times to the distributed cache.
        at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$8.apply(Client.scala:608)
        at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$8.apply(Client.scala:599)
        at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
        at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11.apply(Client.scala:599)

解决方案

# 原因:
sparkOpts=--conf spark.yarn.jars=hdfs:///user/root/share/lib/lib_20180820162943/spark2/scala-compile.jar 
和 
oozie.action.sharelib.for.spark=spark2
# 重复引用出错
将 oozie.action.sharelib.for.spark 中需要引用的包都由--conf来引用即可!
©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页