spark-2.4.4完全分布式搭建

• 下载spark

 wget http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz

• 安装

tar -zxvf spark-2.4.4-bin-hadoop2.7.tgz   #解压
mv spark-2.4.4-bin-hadoop2.7 spark-2.4.4  #改名

• 修改spark-env.sh

export JAVA_HOME=/usr/local/jdk1.8.0_111
export SPARK_MASTER_IP=mini1
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/root/apps/hadoop-3.1.2/etc/hadoop

• 修改slaves

mini1
mini2
mini3

• 把spark安装包分发给其他节点

scp -r spark-2.4.4 mini2:`pwd`
scp -r spark-2.4.4 mini3:`pwd`

• 启动

sbin/start-all.sh