大数据集群搭建（十一）——Spark-2.3.4安装

Spark-2.3.4安装

节点	Master	Worker	Worker
dn1	★
dn2		★
dn3			★

1. 下载spark-2.3.4-bin-hadoop2.7.tgz压缩包
2. 将文件上传到dn1节点的~/pkg目录下
3. 执行以下命令，分别在dn1、dn2、dn3三台节点安装

#解压到/opt目录
cd ~/pkg
tar -xzvf spark-2.3.4-bin-hadoop2.7.tgz -C /opt

#创建spark的软连接
cd /opt
ln -sf spark-2.3.4-bin-hadoop2.7/ spark

4. 配置spark

#配置Worker节点
cp /opt/spark/conf/slaves.template /opt/spark/conf/slaves
vim /opt/spark/conf/slaves

#删除localhost，添加以下内容
dn2
dn3

#配置Master节点
cp /opt/spark/conf/spark-env.sh.template /opt/spark/conf/spark-env.sh
vim /opt/spark/conf/spark-env.sh

#添加以下内容
export SPARK_MASTER_HOST=dn1
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=3g

#将解压后的spark目录分发到dn2和dn3
cd /opt
scp -r spark dn2:`pwd`
scp -r spark dn3:`pwd`

5. 启动spark

/opt/spark/sbin/start-all.sh

6. 搭建spark提交任务的客户端，将spark目录发送到目标节点即可，我们选择nn1

scp -r spark nn1:`pwd`

7. 配置spark运行在Yarn上

#进入nn1中
vim /opt/spark/conf/spark-env.sh
#添加以下内容
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

8. 提交sparkPi任务测试

#基于Standalone提交sparkPi任务
/opt/spark/bin/spark-submit --master spark://dn1:7077 --class org.apache.spark.examples.SparkPi /opt/spark/examples/jars/spark-examples_2.11-2.3.4.jar  100

#基于Yarn提交sparkPi任务
/opt/spark/bin/spark-submit --master yarn --class org.apache.spark.examples.SparkPi /opt/spark/examples/jars/spark-examples_2.11-2.3.4.jar  100