Hive on Spark配置
一、在Hive所在节点部署Spark
首先部署Spark-Yarn模式,配置SPARK_HOME的环境变量。
详见:https://blog.csdn.net/weixin_42796403/article/details/111499940
1)上传并解压解压spark-3.0.0-bin-hadoop3.2.tgz
[hadoop@hadoop102 software]$ tar -zxvf spark-3.0.0-bin-hadoop3.2.tgz -C /opt/module/
[hadoop@hadoop102 software]$ mv /opt/module/ spark-3.0.0-bin-hadoop3.2 /opt/module/spark
2)配置SPARK_HOME环境变量
[hadoop@hadoop102 software]$ sudo vim /etc/profile.d/my_env.sh
添加如下内容
export SPARK_HOME=/opt/module/spark
export PATH=$PATH:$SPARK_HOME/bin
source 使其生效
[hadoop@hadoop102 software]$ source /etc/profile.d/my_env.sh
3)新建spark配置文件
[hadoop@hadoop102 software]$ vim /opt/module/hive/conf/spark-defaults.conf
添加如下内容
spark.master yarn
spark.eventLog.enabled true
spark.eventLog.dir hdfs://hadoop102:8020/spark-history
spark.executor.memory 1g
spark.driver.memory 1g
4)在HDFS创建如下路径
[hadoop@hadoop102 software]$ hadoop fs -mkdir /spark-history
二、向HDFS上传Spark纯净版jar包
1)上传并解压spark-3.0.0-bin-without-hadoop.tgz
[hadoop@hadoop102 software]$ tar -zxvf /opt/software/spark/spark-3.0.0-bin-without-hadoop.tgz
2)上传Spark纯净版jar包到HDFS
[hadoop@hadoop102 software]$ hadoop fs -mkdir /spark-jars
[hadoop@hadoop102 software]$ hadoop fs -put spark-3.0.0-bin-without-hadoop/jars/* /spark-jars
三、修改hive-site.xml文件
[hadoop@hadoop102 ~]$ vim /opt/module/hive/conf/hive-site.xml
添加如下内容
<!--Spark依赖位置-->
<property>
<name>spark.yarn.jars</name>
<value>hdfs://hadoop102:8020/spark-jars/*</value>
</property>
<!--Hive执行引擎-->
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
四、Hive on Spark测试
1)启动hive客户端
[hadoop@hadoop102 hive]$ bin/hive
2)创建一张测试表
hive (default)> create table student(id int, name string);
3)通过insert测试效果
hive (default)> insert into table student values(1,'abc');
若结果如下,则说明配置成功