spark sql读取映射hbase数据的hive外部表报错

集群环境CDH5.8.0 / spark2.1.0

我们用执行以下命令报错：

spark2-submit --master yarn --class com.test.hive.SparkReadHbaseTest ./dacproject.jar 'SELECT count(*) FROM test' 'hdfs:///user/test'

其中test表是从HBASE映射过来的表

报错信息如下：
Exception in thread “main” java.lang.RuntimeException:
java.lang.ClassNotFoundException:org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat

查找网上方法，缺少包：
hbase-site.xml
hbase-protocol-1.2.0-cdh5.8.0.jar
hbase-client-1.2.0-cdh5.8.0.jar
hbase-common-1.2.0-cdh5.8.0.jar
hbase-server-1.2.0-cdh5.8.0.jar
hive-hbase-handler-1.1.0-cdh5.8.0.jar
metrics-core-2.2.0.jar
于是添加后报错：

查找后发现少了
htrace-core-3.2.0-incubating.jar

spark2-submit \
--master local[2] \
--driver-class-path /etc/hbase/conf/hbase-site.xml:/opt/cloudera/parcels/CDH/jars/hbase-protocol-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hbase-client-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hbase-common-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hbase-server-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hive-hbase-handler-1.1.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH/jars/htrace-core-3.2.0-incubating.jar \
--class com.lhx.hive.SparkReadHbaseTest \
./dacproject.jar 'SELECT * FROM test' 'hdfs:///user/test'

最终执行成功！
注：以上是local模式运行，如果要在yarn模式运行，需要每台集群都执行命令：

cp /etc/hbase/conf/hbase-site.xml /opt/cloudera/parcels/SPARK2/lib/spark2/conf
cp /opt/cloudera/parcels/CDH/jars/hbase-protocol-1.2.0-cdh5.8.0.jar cp /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hbase-client-1.2.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hbase-common-1.2.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hbase-server-1.2.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hive-hbase-handler-1.1.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/metrics-core-2.2.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/htrace-core-3.2.0-incubating.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars

最终才算解决问题。