Spark cluster 安装

使用 Ubuntu 机器做主服务

Master
192.168.50.15 (Ubuntu)

Worker 1
192.168.50.15 (Ubuntu)
Worker 2
192.168.50.16 (Windows 10)

先安装 JDK 和下载 Apache Spark

sudo apt-get install openjdk-11-jdk
wget https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
tar xvf spark-3.1.2-bin-hadoop3.2.tgz
sudo mv spark-3.1.2-bin-hadoop3.2 /opt/spark

设置 SPARK_HOME

vi ~/.profile

export SPARK_HOME=/opt/spark

source ~/.profile

设置 服务器 IP

cd /opt/spark/conf
cp spark-env.sh.template spark-env.sh

SPARK_LOCAL_IP=192.168.50.15
SPARK_MASTER_HOST=192.168.50.15

起动 Master 和 Worker 1

sudo /opt/spark/sbin/start-master.sh
sudo /opt/spark/sbin/start-worker.sh spark://192.168.50.15:7077

Windows 下载 Apache Spark 和 Hadoop winutils.exe 放到 e.g. D:\app

https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin

设置环境变量 SPARK_HOME, HADOOP_HOME 和 PATH

SPARK_HOME
D:\app\spark
PATH add %SPARK_HOME%\bin


HADOOP_HOME
D:\app\hadoop
PATH add %HADOOP_HOME%\bin

打开命令字元

spark-class org.apache.spark.deploy.worker.Worker spark://192.168.50.15:7077

查看 Spark Master UI

使用 Ctrl-C 来停止 Windows Worker 2

停止 Worker 1 和 Master

sudo /opt/spark/sbin/stop-worker.sh
sudo /opt/spark/sbin/stop-master.sh