spark-standalone安装教程

安装环境：CentOS-7、jdk

软件版本：spark-2.3.4-bin-hadoop2.6

资源路径：百度网盘 请输入提取码 

提取码：zzzz

1、将安装包上传到 /opt 目录下

2、解压安装包

tar -zxvf spark-2.3.4-bin-hadoop2.6.tgz -C soft/spark-standby/ --strip-components 1

3、将 /opt/soft/spark-standby/conf/slaves.template 拷贝一份改名为slaves，并编辑添加自己的主机ip或主机映射

cp /opt/soft/spark-standby/conf/slaves.template /opt/soft/spark-standby/conf/slaves

vim /opt/soft/spark-standby/conf/slaves

4、将 /opt/soft/spark-standby/conf/spark-env.sh.template 拷贝一份改名为 /opt/soft/spark-standby/conf/spark-env.sh，并添加如下配置信息

cp /opt/soft/spark-standby/conf/spark-env.sh.template /opt/soft/spark-standby/conf/spark-env.sh

vim /opt/soft/spark-standby/conf/spark-env.sh

export SPARK_MASTER_HOST=hadoop101
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=3g
export SPARK_MASTER_WEBUI_PORT=8888

5、编辑 /opt/soft/spark-standby/sbin/spark-config.sh 文件，并添加JAVA_HOME路径

vim /opt/soft/spark-standby/sbin/spark-config.sh

export JAVA_HOME=/opt/soft/jdk180

6、配置历史服务器，拷贝一份 /opt/soft/spark-standby/conf/spark-defaults.conf.template，为/opt/soft/spark-standby/conf/spark-defaults.conf，并编辑

cp /opt/soft/spark-standby/conf/spark-defaults.conf.template /opt/soft/spark-standby/conf/spark-defaults.conf

vim /opt/soft/spark-standby/conf/spark-defaults.conf

 注意！！！下图 hdfs://hadoop101:9000/spark-log-standby 这个目录必须事先创建好

7、继续编辑 /opt/soft/spark-standby/conf/spark-env.sh，添加如下内容

vim /opt/soft/spark-standby/conf/spark-env.sh

export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retained
Applications=30 -Dspark.history.fs.logDirectory=hdfs://hadoop101:9000/spark-log-standby"

8、启动hdfs，把之前填写的日志保存的目录创建好

start-dfs.sh

hdfs dfs -mkdir -p /spark-standby-standby

9、启动spark

/opt/soft/spark-standby/sbin/start-all.sh

10、开启日志服务器

/opt/soft/spark-standby/sbin/start-hostory-server.sh

11、打开 maser web UI

http://192.168.1.101:8888

12、打开日志服务器 web UI

http://192.168.1.101:18080

13、停止spark

/opt/soft/spark-standby/sbin/stop-all.sh