Ubuntu18.04 部署Spark集群
1. Spark集群部署
下载地址:http://spark.apache.org/downloads.html
集群配置
master | 192.168.0.130 | Master |
slave1 | 192.168.0.131 | Worker |
slave2 | 192.168.0.132 | Worker |
1.1 安装Spark
进入/opt目录,解压 spark-2.3.3-bin-hadoop2.7.tgz
sudo tar -zxvf spark-2.3.3-bin-hadoop2.7.tgz /opt
sudo mv /opt/spark-2.3.3-bin-hadoop2.7 /opt/spark
修改环境变量
sudo gedit /etc/profile
export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH
刷新环境变量
source /etc/profile
1.2 配置Spark
1.2.1 配置spark-env.sh
进入SPARK_HOME下的 conf 目录, 拷贝 spark-env.sh.template 到 spark-env.sh
sudo cp spark-env.sh.template spark-env.sh
编辑 spark-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export SPARK_MASTER_IP=master
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
export HIVE_HOME=/opt/hive
1.2.2 配置 slaves
拷贝slaves.template 到 slaves
sudo cp slaves.template slaves
编辑slaves
slave1
slave2
1.2.3 将上述所有文件复制一份到各节点
1.2.4 启动Spark集群
cd /opt/spark/sbin/
./start-all.sh
启动成功,访问http://master:8080/