Hive on Spark配置

Hive on Spark配置

一、在Hive所在节点部署Spark

首先部署Spark-Yarn模式，配置SPARK_HOME的环境变量。
详见：https://blog.csdn.net/weixin_42796403/article/details/111499940

1）上传并解压解压spark-3.0.0-bin-hadoop3.2.tgz

[hadoop@hadoop102 software]$ tar -zxvf spark-3.0.0-bin-hadoop3.2.tgz -C /opt/module/ 
[hadoop@hadoop102 software]$ mv /opt/module/ spark-3.0.0-bin-hadoop3.2 /opt/module/spark

2）配置SPARK_HOME环境变量

[hadoop@hadoop102 software]$ sudo vim /etc/profile.d/my_env.sh

添加如下内容

export SPARK_HOME=/opt/module/spark
export PATH=$PATH:$SPARK_HOME/bin

source 使其生效

[hadoop@hadoop102 software]$ source /etc/profile.d/my_env.sh

3）新建spark配置文件

[hadoop@hadoop102 software]$ vim /opt/module/hive/conf/spark-defaults.conf

添加如下内容

spark.master                               yarn
spark.eventLog.enabled                   true
spark.eventLog.dir                        hdfs://hadoop102:8020/spark-history
spark.executor.memory                    1g
spark.driver.memory					   1g

4）在HDFS创建如下路径

[hadoop@hadoop102 software]$ hadoop fs -mkdir /spark-history

二、向HDFS上传Spark纯净版jar包

1）上传并解压spark-3.0.0-bin-without-hadoop.tgz

[hadoop@hadoop102 software]$ tar -zxvf /opt/software/spark/spark-3.0.0-bin-without-hadoop.tgz

2）上传Spark纯净版jar包到HDFS

[hadoop@hadoop102 software]$ hadoop fs -mkdir /spark-jars
[hadoop@hadoop102 software]$ hadoop fs -put spark-3.0.0-bin-without-hadoop/jars/* /spark-jars

三、修改hive-site.xml文件

[hadoop@hadoop102 ~]$ vim /opt/module/hive/conf/hive-site.xml

添加如下内容

<!--Spark依赖位置-->
<property>
    <name>spark.yarn.jars</name>
    <value>hdfs://hadoop102:8020/spark-jars/*</value>
</property>
  
<!--Hive执行引擎-->
<property>
    <name>hive.execution.engine</name>
    <value>spark</value>
</property>

四、Hive on Spark测试

1）启动hive客户端

[hadoop@hadoop102 hive]$ bin/hive

2）创建一张测试表

hive (default)> create table student(id int, name string);

3）通过insert测试效果

hive (default)> insert into table student values(1,'abc');

若结果如下，则说明配置成功