在利用sbt-assmebly对Spark应用打包时,提示如下错误:
java.lang.RuntimeException: deduplicate: different file contents found in the following:
[error] C:\Users\didi\.ivy2\cache\javax.inject\javax.inject\jars\javax.inject-1.jar:javax/inject/Inject.class
[error] C:\Users\didi\.ivy2\cache\org.glassfish.hk2.external\javax.inject\jars\javax.inject-2.4.0-b34.jar:javax/inject/Inject.class
[error] deduplicate: different file contents found in the following:
[error] C:\Users\didi\.ivy2\cache\javax.inject\javax.inject\jars\javax.inject-1.jar:javax/inject/Named.class
[error] C:\Users\didi\.ivy2\cache\org.glassfish.hk2.external\javax.inject\jars\javax.inject-2.4.0-b34.jar:javax/inject/Named.class
从内容可以看出,问题出现在有重复的class文件,所以解决的思路也很简单,哪里重复就合并哪里!
具体的做法是:
1. 找到出错的包名;
2. 将包名替换为PathList(x, y, xs @ _*),添加在“case”列表中;
以上面的错误为例:
1. 找到包名,“javax/inject”;
2. 替换为PathList(“javax”, “inject”),添加在“case”列表中,如下:
lazy val root = (project in file("."))
.settings(
sparkSettings,
name := "spark-assembly",
mainClass in assembly := Some("com.mirana.spark.ExampleApp"),
assemblyJarName in assembly := "scala-spark-final.jar",
assemblyMergeStrategy in assembly := {
// 需要合并的内容
case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
// 保持不变
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
)
完整的配置内容及项目,请参见我的git示例-spark-assembly。