下载 spark-2.4.0-bin-hadoop2.7.tgz
链接:https://pan.baidu.com/s/1dlZlEcvwPck1JpSdBbXyYw
提取码:3y22
解压 spark-2.4.0-bin-hadoop2.7.tgz
[hadoop@spark1 softwares]$ tar spark-2.4.0-bin-hadoop2.7.tgz -C /usr/local/modules/
cd /usr/local/modules/spark-2.4.0-bin-hadoop2.7/conf
文件重命名
mv spark-env.sh.template spark-env.sh
mv slaves.template slaves
配置 spark-env.sh
vim spark-env.sh
export SCALA_HOME=/usr/local/modules/scala-2.10.4
export JAVA_HOME=/usr/local/modules/jdk1.8.0_201
export HADOOP_HOME=/usr/local/modules/hadoop-2.7.7
export HADOOP_CONF_DIR=/usr/local/modules/hadoop-2.7.7/etc/hadoop
export SPARK_HOME=/usr/local/modules/spark-2.4.0-bin-hadoop2.7
export SPARK_MASTER_IP=spark1
配置 slaves
vim slaves
spark2
spark3
复制hive-site.xml 到 /usr/local/modules/spark-2.4.0-bin-hadoop2.7/conf 下
cp hive-site.xml /usr/local/modules/spark-2.4.0-bin-hadoop2.7/conf
cp spark到其他服务器
scp -r spark-2.4.0-bin-hadoop2.7 hadoop@spark1:/usr/local/modules
scp -r spark-2.4.0-bin-hadoop2.7 hadoop@spark2:/usr/local/modules
启动hive的metastore服务
hive --service metastore &
启动spark 进程 master 和 slaves
start-all.sh
spark1的进程
spark2的进程
spark3的进程
在浏览器上可以看 master 的 8080端口,如下图,则spark启动成功