spark-3.0.0安装

spark-3.0.0安装

官网下载

https://spark.apache.org/downloads.html

将spark.tgz包上传到虚拟机

如图，此版本需要下载scala 2.12.X

解压tgz包到指定目录

tar -zxvf spark-3.0.0-bin-hadoop3.2.tgz -C /new/software/
cd /new/software/spark-3.0.0-bin-hadoop3.2/conf
cp spark-env.sh.template spark-env.sh

vim spark-env.sh

export SCALA_HOME=/new/software/scala-2.12.4
export JAVA_HOME=/new/software/jdk1.8.0_141
export HADOOP_HOME=/new/software/hadoop-3.3.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
SPARK_MASTER_IP=yts1
SPARK_LOCAL_DIRS=/new/software/hadoop-3.3.0
SPARK_DRIVER_MEMORY=512M

cp slaves.template slaves

vim slaves

yts1
yts2
yts3

vim /etc/profile

export SPARK_HOME=/new/software/spark-3.0.0-bin-hadoop3.2
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

source /etc/profile

将spark目录以及/etc/profile同步到其他机器

运行官方样类

run-example SparkPi 10

结果