Ambari集群启用Hive on spark

环境信息

Ambari 2.6.1+HDP 2.6.5(hive 单独升级到了2.3.2)+spark 2.0

参考资料

https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark:+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive.

https://stackoverflow.com/questions/55422935/hive-on-spark-error-java-lang-nosuchfielderror-spark-rpc-server-address

https://www.cnblogs.com/lyy-blog/p/9598433.html

配置步骤

1. 将spark 依赖包 软链接到hive目录

ln -s /usr/hdp/2.6.5.0-292/spark2/jars/scala-library-2.11.8.jar /usr/hdp/2.6.5.0-292/hive/lib
ln -s /usr/hdp/2.6.5.0-292/spark2/jars/spark-core_2.11-2.3.0.2.6.5.0-292.jar /usr/hdp/2.6.5.0-292/hive/lib
ln -s /usr/hdp/2.6.5.0-292/spark2/jars/spark-network-common_2.11-2.3.0.2.6.5.0-292.jar /usr/hdp/2.6.5.0-292/hive/lib
ln -s /usr/hdp/2.6.5.0-292/spark2/jars/spark-unsafe_2.11-2.3.0.2.6.5.0-292.jar /usr/hdp/2.6.5.0-292/hive/lib

2. 在custom hive-site.xml配置里添加如下配置

<property>
  <name>spark.yarn.jars</name>
  <value>hdfs:///spark-jars/*</value>
</property>
<property>
  <name>spark.master</name>
  <value>yarn-cluster</value>
</property>
<property>
  <name>spark.sql.hive.metastore.version</name>
  <value>2.3.2</value>
</property>
<property>
  <name>spark.sql.hive.metastore.jars </name>
  <value>/usr/hdp/2.6.5.0-292/hive/lib/*:/usr/hdp/2.6.5.0-292/hadoop/client/*
</value>
</property>

3. 将spark依赖的jar包上传到hdfs，避免加载本地jar包

hdfs dfs -mkdir /spark-jars
hdfs dfs -put /usr/hdp/2.6.5.0-292/spark2/jars/*.jar /spark-jars/
hdfs dfs -rm /spark-jars/*hive*1.2.1*

测试

beeline -u "jdbc:hive2://ip.ip.ip.ip:10000/"
set hive.execution.engine=spark;
select count(1) from hive_on_spark_test;

建议

hive on spark 目前的兼容性十分有限，官方给出了以下兼容性表，后续演进也尚不明确，建议不要采用这种方案，能直接用sparksql 就用sparksql吧