Spark安装与配置

环境

操作系统：Centos 7
Hadoop版本：2.9.2
JDK版本：1.8.0_221
Spark版本：2.4.7
集群规划：

主机名	IP
master	192.168.1.121
slave1	192.168.1.122
slave2	192.168.1.123

下载

下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.7/

安装

首先将下载的压缩包解压到home/apps/目录下

[root@master dev]# mkdir -p /home/apps/
[root@master dev]# tar -zxvf  spark-2.4.7-bin-hadoop2.7.tgz -C /home/apps
[root@master dev]# cd /home/apps/
[root@master apps]# ll
total 24
drwxr-xr-x.  7 root   root   4096 Jan 23 10:33 flume
drwxr-xr-x.  7 root   root   4096 Jan 21 17:13 hbase
drwxr-xr-x.  7 root   root   4096 Jan 22 15:04 kafka
drwxr-xr-x. 15 centos centos 4096 Jan 23 14:38 spark-2.4.7-bin-hadoop2.7
drwxr-xr-x.  9 centos centos 4096 Dec 19  2017 sqoop
drwxr-xr-x.  8 root   root   4096 Jan 15 20:58 zookeeper

将文件名改为spark

[root@master dev]# mv spark-2.4.7-bin-hadoop2.7 spark

配置

spark需要配置spark-env.sh、slaves文件

复制spark-env.sh.template文件为spark-env.sh
复制slaves.template文件为slaves

[root@master apps]# cd spark/conf/
[root@master conf]# cp spark-env.sh.template spark-env.sh 
[root@master conf]# cp slaves.template slaves

配置spark文件

[root@master conf]# vi spark-env.sh

在其末尾添加以下配置

#配置JAVA_HOME，注意更改路径
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_221/

#设置Master的主机名
export SPARK_MASTER_HOST=master

#每一个Worker最多可以使用的内存
export SPARK_WORKER_MEMORY=1g

#每一个Worker最多可以使用的cpu core的个数
export SPARK_WORKER_CORES=1

#提交Application的端口
export SPARK_MASTER_PORT=7077

#spark任务有很大可能性需要去HDFS上读取文件，所以需要配置，注意更改路径  
export SPARK_CONF_DIR=/home/hadoop/hadoop-2.9.2/etc/hadoop

配置slaves文件

[root@master conf]# vi slaves

在其末尾添加每个节点的主机名

master
slave1
slave2

配置环境变量（每个节点都需要配置环境变量）

[root@master conf]# vi /etc/profile

在其末尾添加以下配置

#spark的安装路径
export SPARK_HOME=/home/apps/spark
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

使用source命令使添加的环境变量立即生效

[root@master conf]# source /etc/profile

同样的操作，在slave1和slave2上面配置

注意：因为Hadoop也配置了环境变量，它们的名字一样，所以需要更改Hadoop或者spark启动文件名，如果不更改文件名，那么很有可能将启动不成功

更改sbin目录下的启动文件名

[root@master conf]# cd /home/apps/spark/sbin
[root@master sbin]# ll
 total 92
-rwxr-xr-x. 1 centos centos 2803 Sep  8 13:48 slaves.sh
-rwxr-xr-x. 1 centos centos 1429 Sep  8 13:48 spark-config.sh
-rwxr-xr-x. 1 centos centos 5689 Sep  8 13:48 spark-daemon.sh
-rwxr-xr-x. 1 centos centos 1262 Sep  8 13:48 spark-daemons.sh
-rwxr-xr-x. 1 centos centos 1274 Sep  8 13:48 start-history-server.sh
-rwxr-xr-x. 1 centos centos 2050 Sep  8 13:48 start-master.sh
-rwxr-xr-x. 1 centos centos 1877 Sep  8 13:48 start-mesos-dispatcher.sh
-rwxr-xr-x. 1 centos centos 1423 Sep  8 13:48 start-mesos-shuffle-service.sh
-rwxr-xr-x. 1 centos centos 1279 Sep  8 13:48 start-shuffle-service.sh
-rwxr-xr-x. 1 centos centos 3151 Sep  8 13:48 start-slave.sh
-rwxr-xr-x. 1 centos centos 1527 Sep  8 13:48 start-slaves.sh
-rwxr-xr-x. 1 centos centos 1190 Sep  8 13:48 start-all.sh
-rwxr-xr-x. 1 centos centos 1857 Sep  8 13:48 start-thriftserver.sh
-rwxr-xr-x. 1 centos centos 1056 Sep  8 13:48 stop-history-server.sh
-rwxr-xr-x. 1 centos centos 1080 Sep  8 13:48 stop-master.sh
-rwxr-xr-x. 1 centos centos 1227 Sep  8 13:48 stop-mesos-dispatcher.sh
-rwxr-xr-x. 1 centos centos 1084 Sep  8 13:48 stop-mesos-shuffle-service.sh
-rwxr-xr-x. 1 centos centos 1067 Sep  8 13:48 stop-shuffle-service.sh
-rwxr-xr-x. 1 centos centos 1557 Sep  8 13:48 stop-slave.sh
-rwxr-xr-x. 1 centos centos 1064 Sep  8 13:48 stop-slaves.sh
-rwxr-xr-x. 1 centos centos 1478 Sep  8 13:48 stop-all.sh
-rwxr-xr-x. 1 centos centos 1066 Sep  8 13:48 stop-thriftserver.sh
[root@master sbin]# mv start-all.sh start-spark-all.sh
[root@master sbin]# mv stop-all.sh stop-spark-all.sh

将master上面的spark文件夹传输到其他节点上

[root@master sbin]# scp -r /home/apps/spark/ root@slave1:/home/apps/
[root@master sbin]# scp -r /home/apps/spark/ root@slave2:/home/apps/

启动

spark启动命令

[root@master sbin]# start-spark-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/apps/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /home/apps/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out
slave3: starting org.apache.spark.deploy.worker.Worker, logging to /home/apps/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave3.out
master: starting org.apache.spark.deploy.worker.Worker, logging to /home/apps/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out

启动成功

Spark的交互式界面

[root@master sbin]# cd /home/apps/bin/
[root@master bin]# ./spark-shell
21/01/23 15:30:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/01/23 15:30:50 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark context Web UI available at http://master:4041
Spark context available as 'sc' (master = local[*], app id = local-1611387050873).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.7
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_221)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

结语：大数据Hadoop笔记 Spark 安装与配置