spark完全分布式集群

spark完全分布式集群

解压spark压缩包并修改名字

tar -zxvf /opt/software/spark-3.2.1-bin-hadoop2.7.tgz -C /opt/module/
mv /opt/module/spark-3.2.1-bin-hadoop2.7/ /opt/module/spark-3.2.1/

切换至spark安装目录的conf文件夹下

cd /opt/module/spark-3.2.1/conf/

配置spark-env.sh文件

cp spark-env.sh.template spark-env.sh
vi spark-env.sh

添加下面内容至spark-env.sh文件
export JAVA_HOME=/opt/module/jdk1.8.0_361
export HADOOP_CONF_DIR=/opt/module/hadoop-3.1.4/etc/hadoop
export SPARK_MASTER_IP=master
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_MEMORY=1024m
export SPARK_WORKER_CORES=1
export SPARK_EXECUTOR_MEMORY=1024m
export SPARK_EXECUTOR_CORES=1
export SPARK_WORKER_INSTANCES=1

配置workers文件

cp workers.template workers
vi workers

替换内容至workers文件
slave1
slave2

配置spark-defaults.conf文件

cp spark-defaults.conf.template spark-defaults.conf
vi spark-defaults.conf

添加下面内容至spark-defaults.conf文件
spark.master                     spark://master:7077
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://master:9000/spark-logs
spark.history.fs.logDirectory    hdfs://master:9000/spark-logs

分发文件至从机

scp -r /opt/module/spark-3.2.1/ root@slave1:/opt/module/
scp -r /opt/module/spark-3.2.1/ root@slave2:/opt/module/

启动spark集群前，需要启动hadoop并创建spark-logs目录

cd $HADOOP_HOME
./sbin/start-all.sh
hdfs dfs -mkdir /spark-logs

启动spark集群

cd /opt/module/spark-3.2.1/sbin/
./start-all.sh
./start-history-server.sh

启动shell

/opt/module/spark-3.2.1/bin/spark-shell

scala安装

解压缩

tar -zxvf /opt/software/scala-2.11.12.tgz -C /opt/module/

配置环境变量

vi /etc/profile

添加：
# SCALA_HOME
export SCALA_HOME=/opt/module/scala-2.11.12
export PATH=$PATH:$SCALA_HOME/bin

刷新环境变量并查看scala版本号

source /etc/profile
scala -version

启动Scala

scala