问题描述:
hbase 1.5在spark2.x中使用hbase-spark 依赖访问Hbase的时候,编写的程序无法完成编译,在编译过程中报入下错误
Error:scalac: missing or invalid dependency detected while loading class file 'HBaseContext.class'.
Could not access type Logging in package org.apache.spark,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'HBaseContext.class' was compiled against an incompatible version of org.apache.spark.
编译出错,无法正常打包
问题解决
经过查看资料找到HBase官网的一个JIRA,地址:
https://issues.apache.org/jira/browse/HBASE-16428
JIRA中找到在代码中引用了HBaseContext时,使用Spark2编译Spark应用程序将会失败,因为HBaseContext模块引用了org.apache.spark.Logging。在Spark2中,由于Logging被移动到一个私有的包下导致。同样也有Spark2的一个JIRA说明该问题:
https://issues.apache.org/jira/browse/SPARK-13928
为了完成编译,这里Fayson使用如下做法解决该问题:
1.在自己的工程下创建一个org.apache.spark的包
2.在org.apache.spark包下创建一个Trait类型的Logging.scala类型,该类的内容通过Spark2源码找到
重新编译即可解决