Hive On Spark（切换计算引擎为Spark）

百度网盘下载地址（该系列版本 平台搭建全套安装包）：
链接： https://pan.baidu.com/s/11HyrxVKevgvJc-rI6bFLjA 提取码：sybk

将Hive默认的MapReduce计算引擎替换成Spark
本文章基于已经安装配置好完成元数据初始化的Hive

关闭Yarn内存检查

修改hadoop的yarn-site.xml配置文件

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

修改hadoop配置后需要分发并重启集群

上传Spark安装包并解压

注意要选择版本对应的.避免出现版本兼容性问题
tar -zxvf spark-3.0.0-bin-hadoop3.2.tgz -C /opt/module/
mv /opt/module/spark-3.0.0-bin-hadoop3.2/ /opt/module/spark

配置Spark环境变量

export SPARK_HOME=/opt/module/spark

export PATH=$PATH:$SPARK_HOME/bin

source /etc/profile.d/myenv.sh

存储Spark默认参数 spark-defaults.conf

spark.master                   yarn
spark.eventLog.enabled          true
spark.eventLog.dir              hdfs://master:9000/spark/logs
spark.executor.memory           1g
spark.driver.memory             1g

向HDFS上传纯净版的spark的jar包

hdfs dfs -put /opt/module/spark/spark-3.0.0-bin-without-hadoop/jars/* /spark/jars

​

#事先创建要用到的HDFS目录

hdfs dfs -mkdir -p /spark/jars

hdfs dfs -mkdir -p /spark/logs

修改hive-site.xml配置文件

<!-- Hive on Spark-->
<property>
    <name>spark.yarn.jars</name>
    <value>hdfs://master:9000/spark/jars</value>
</property>
<property>
    <name>hive.execution.engine</name>
    <value>spark</value>
</property>

修改spark-env.sh配置文件

export JAVA_HOME=/opt/module/jdk
export YARN_CONF_DIR=/opt/module/hadoop/etc/hadoop
export HADOOP_CONF_DIR=/opt/module/hadoop/etc/hadoop
export SPARK_HOME=/opt/module/spark
export SPARK_DIST_CLASSPATH=$(hadoop classpath)

SPARK_DAEMON_JAVA_OPTS="
-Dspark.deploy.recoveryMode=ZOOKEEPER
-Dspark.deploy.zookeeper.url=master:2181,slave1:2181,slave2:2181
-Dspark.deploy.zookeeper.dir=/spark/zookeeper"
​
SPARK_HISTORY_OPTS="
-Dspark.history.fs.logDirectory=hdfs://master:9000/spark/logs
-Dspark.history.fs.cleaner.enabled=true"

hadoop classpath 需要在环境变量中配置好

测试

create database tmp;
use tmp;
create table student (id int, name string);
insert into table student values(1,'abc');

报错org.apache.hadoop.hive.ql.parse.SemanticException:Failed to get a spark session:
查看export SPARK_DIST_CLASSPATH=$(hadoop classpath)是否配置成功