整体概述
Spark3.2.0 在Centos7 部署安装教程
版本选择
-
jdk-8u202
重要:Spark3.2.0 版本官网依赖jdk8,8u201之前的版本已经被宣布弃用了;所以jdk 选择8u202(java最后一个免费版) -
spark-3.2.0-bin-hadoop3.2.tgz
由于我们是Standalone模式部署,所以不用部署Hadoop集群 -
Scala 2.13.7
安装步骤
整体部署概览
主机名称 | ip地址 | 节点 | 备注 |
---|---|---|---|
spark01.bit | 192.168.223.3 | master | |
spark02.bit | 192.168.223.4 | slave | |
spark03.bit | 192.168.223.5 | slave | |
spark04.bit | 192.168.223.6 | slave |
- 修改hosts 信息,添加下列信息
[root@spark01 ~]# vim /etc/hosts
加入
192.168.223.3 spark01.bit
192.168.223.4 spark02.bit
192.168.223.5 spark03.bit
192.168.223.6 spark04.bit
- 分发hosts文件到各个节点主机
[root@spark01 ~]# scp /etc/hosts root@192.168.223.4:/etc/hosts
[root@spark01 ~]# scp /etc/hosts root@192.168.223.5:/etc/hosts
[root@spark01 ~]# scp /etc/hosts root@192.168.223.6:/etc/hosts
- 生成ras 公私钥对,向各个主机分发自己的秘钥,用于SSH免密登陆
[root@spark01 ~]# ssh-keygen -t ras
[root@spark01 ~]# ssh-copy-id -i spark02.bit
每台主机都要登陆执行,想另外三台分发自己秘钥,执行方式跟上边一致,就不一一演示了
- 分别解压 Jdk、spark、scala 到 /usr/local/ 目录下
向各个主机分发软件,示例:
[root@spark01 local]# scp -rq /usr/local/java root@spark02.bit:/usr/local/
[root@spark01 local]# scp -rq /usr/local/spark-3.2.0-bin-hadoop3.2 root@spark02.bit:/usr/local
[root@spark01 local]# scp -rq /usr/local/scala-2.13.7 root@spark03.bit:/usr/local
- 修改主机 /etc/profiles
[root@spark01 local]# vim /etc/profile
底部加入
export SPARK_HOME=/usr/local/spark-3.2.0-bin-hadoop3.2
export JAVA_HOME=/usr/local/java/jdk1.8.0_202
export SCALA_HOME=/usr/local/scala-2.13.7
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:${JAVA_HOME}/bin
export PATH=$PATH:$JAVA_HOME/bin:${SPARK_HOME}/sbin:${SPARK_HOME}/bin:${SCALA_HOME}/bin
- 环境变量生效
[root@spark01 local]# source /etc/profile
- 向其他各主机分发环境变量配置文件
[root@spark01 ~]# scp /etc/profile root@192.168.223.4:/etc/profile
[root@spark01 ~]# scp /etc/profile root@192.168.223.5:/etc/profile
[root@spark01 ~]# scp /etc/profile root@192.168.223.6:/etc/profile
- 修改spark 配置文件
[root@spark01 local]# cd /usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 sbin]# vi spark-env.sh
加入
export JAVA_HOME=/usr/local/java/jdk1.8.0_202
export SPARK_HOME=/usr/local/spark-3.2.0-bin-hadoop3.2
修改workers(spark2.0版本,文件名称为:slaves) 文件,加入一下文件
[root@spark01 conf]# vim workers
spark02.bit
spark03.bit
spark04.bit
scp配置到各个从节点
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh root@spark02.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh root@spark03.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh root@spark04.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/spark-env.sh root@spark05.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers root@spark02.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers root@spark03.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers root@spark04.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf/
[root@spark01 conf] scp -r /usr/local/spark-3.2.0-bin-hadoop3.2/conf/workers root@spark05.bit:/usr/local/spark-3.2.0-bin-hadoop3.2/conf
10.启动
[root@spark01 conf]# cd /usr/local/spark-3.2.0-bin-hadoop3.2/sbin/
[root@spark01 sbin]# start-all.sh
启动中:
启动完成:
11.提交task作业测试
[root@spark01 sbin]# spark-submit --class org.apache.spark.examples.SparkPi --master spark://spark01.bit:7077 --num-executors 2 /usr/local/spark-3.2.0-bin-hadoop3.2/examples/jars/spark-examples_2.12-3.2.0.jar
查看计算结果和界面任务