spark完全分布式集群
解压spark压缩包并修改名字
tar -zxvf /opt/software/spark-3.2.1-bin-hadoop2.7.tgz -C /opt/module/
mv /opt/module/spark-3.2.1-bin-hadoop2.7/ /opt/module/spark-3.2.1/
切换至spark安装目录的conf文件夹下
cd /opt/module/spark-3.2.1/conf/
配置spark-env.sh文件
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
添加下面内容至spark-env.sh文件
export JAVA_HOME=/opt/module/jdk1.8.0_361
export HADOOP_CONF_DIR=/opt/module/hadoop-3.1.4/etc/hadoop
export SPARK_MASTER_IP=master
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_MEMORY=1024m
export SPARK_WORKER_CORES=1
export SPARK_EXECUTOR_MEMORY=1024m
export SPARK_EXECUTOR_CORES=1
export SPARK_WORKER_INSTANCES=1
配置workers文件
cp workers.template workers
vi workers
替换内容至workers文件
slave1
slave2
配置spark-defaults.conf文件
cp spark-defaults.conf.template spark-defaults.conf
vi spark-defaults.conf
添加下面内容至spark-defaults.conf文件
spark.master spark://master:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://master:9000/spark-logs
spark.history.fs.logDirectory hdfs://master:9000/spark-logs
分发文件至从机
scp -r /opt/module/spark-3.2.1/ root@slave1:/opt/module/
scp -r /opt/module/spark-3.2.1/ root@slave2:/opt/module/
启动spark集群前,需要启动hadoop并创建spark-logs目录
cd $HADOOP_HOME
./sbin/start-all.sh
hdfs dfs -mkdir /spark-logs
启动spark集群
cd /opt/module/spark-3.2.1/sbin/
./start-all.sh
./start-history-server.sh
启动shell
/opt/module/spark-3.2.1/bin/spark-shell
![](https://i-blog.csdnimg.cn/blog_migrate/a65b41499b7a1615af5afbfb9716499c.png)
scala安装
解压缩
tar -zxvf /opt/software/scala-2.11.12.tgz -C /opt/module/
配置环境变量
vi /etc/profile
添加:
export SCALA_HOME=/opt/module/scala-2.11.12
export PATH=$PATH:$SCALA_HOME/bin
刷新环境变量并查看scala版本号
source /etc/profile
scala -version
启动Scala
scala
![](https://i-blog.csdnimg.cn/blog_migrate/90684f1ef1a8736e61731b5f73bc8177.png)