oozie on spark2:基于Yarn运行Spark2时出现的错误说明(scala does not exist; make sure Spark is built.)

Oozie adds all of the jars in the Oozie Spark sharelib to the DistributedCache such that all jars will be present in the current working directory of the YARN container (as well as in the container classpath). However, this is not quite enough to make Spark 2.0 work, since Spark 2.0 by default looks for the jars in assembly/target/scala-2.11/jars [1] (as if it is a locally built distribution for development) and will not find them in the current working directory.

To fix this, we can set spark.yarn.jars to *.jar so that it finds the jars in the current working directory rather than looking in the wrong place.

©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页