Spark3.2.0 Standalone集群模式部署教程

整体概述

Spark3.2.0 在Centos7 部署安装教程

版本选择

• jdk-8u202
  重要：Spark3.2.0 版本官网依赖jdk8，8u201之前的版本已经被宣布弃用了；所以jdk 选择8u202(java最后一个免费版)

• spark-3.2.0-bin-hadoop3.2.tgz
  由于我们是Standalone模式部署，所以不用部署Hadoop集群

• Scala 2.13.7

安装步骤

整体部署概览

主机名称	ip地址	节点	备注
spark01.bit	192.168.223.3	master
spark02.bit	192.168.223.4	slave
spark03.bit	192.168.223.5	slave
spark04.bit	192.168.223.6	slave

1. 修改hosts 信息，添加下列信息

 [root@spark01 ~]# vim /etc/hosts

加入

192.168.223.3 spark01.bit
192.168.223.4 spark02.bit
192.168.223.5 spark03.bit
192.168.223.6 spark04.bit

2. 分发hosts文件到各个节点主机

[root@spark01 ~]# scp /etc/hosts  root@192.168.223.4:/etc/hosts
[root@spark01 ~]# scp /etc/hosts  root@192.168.223.5:/etc/hosts
[root@spark01 ~]# scp /etc/hosts  root@192.168.223.6:/etc/hosts

3. 生成ras 公私钥对，向各个主机分发自己的秘钥，用于SSH免密登陆

[root@spark01 ~]# ssh-keygen -t ras
[root@spark01 ~]# ssh-copy-id -i  spark02.bit

每台主机都要登陆执行，想另外三台分发自己秘钥，执行方式跟上边一致，就不一一演示了

4. 分别解压 Jdk、spark、scala 到 /usr/local/ 目录下
   向各个主机分发软件，示例：

[root@spark01 local]# scp -rq  /usr/local/java  root@spark02.bit:/usr/local/
[root@spark01 local]# scp -rq  /usr/local/spark-3.2.0-bin-hadoop3.2  root@spark02.bit:/usr/local 
[root@spark01 local]# scp -rq  /usr/local/scala-2.13.7 root@spark03.bit:/usr/local 

5. 修改主机 /etc/profiles

 [root@spark01 local]# vim /etc/profile

底部加入

export SPARK_HOME=/usr/local/spark-3.2.0-bin-hadoop3.2
export JAVA_HOME=/usr/local/java/jdk1.8.0_202
export SCALA_HOME=/usr/local/scala-2.13.7
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:${JAVA_HOME}/bin
export PATH=$PATH:$JAVA_HOME/bin:${SPARK_HOME}/sbin:${SPARK_HOME}/bin:${SCALA_HOME}/bin

6. 环境变量生效

 [root@spark01 local]# source /etc/profile

7. 向其他各主机分发环境变量配置文件

[root@spark01 ~]# scp /etc/profile  root@192.168.223.4:/etc/profile
[root@spark01 ~]# scp /etc/profile  root@192.168.223.5:/etc/profile
[root@spark01 ~]# scp /etc/profile  root@192.168.223.6:/etc/profile

9. 修改spark 配置文件

[root@spark01 local]# cd /usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 sbin]# vi spark-env.sh  

加入

export JAVA_HOME=/usr/local/java/jdk1.8.0_202
export SPARK_HOME=/usr/local/spark-3.2.0-bin-hadoop3.2

修改workers（spark2.0版本，文件名称为：slaves） 文件，加入一下文件

[root@spark01 conf]# vim workers
spark02.bit
spark03.bit
spark04.bit     

scp配置到各个从节点

[root@spark01 conf] scp  -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh   root@spark02.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp  -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh  root@spark03.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp  -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh  root@spark04.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh   root@spark05.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf

[root@spark01 conf] scp  -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers  root@spark02.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp  -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers  root@spark03.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp  -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers  root@spark04.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r  /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers  root@spark05.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf

10.启动

[root@spark01 conf]# cd /usr/local/spark-3.2.0-bin-hadoop3.2/sbin/
[root@spark01 sbin]# start-all.sh 

启动中：
启动完成:

11.提交task作业测试

[root@spark01 sbin]# spark-submit  --class org.apache.spark.examples.SparkPi --master spark://spark01.bit:7077 --num-executors  2 /usr/local/spark-3.2.0-bin-hadoop3.2/examples/jars/spark-examples_2.12-3.2.0.jar

查看计算结果和界面任务