1.安装scala
参考:http://blog.csdn.net/weixin_36104843/article/details/80212517
2.下载sprak
wget http://mirrors.shu.edu.cn/apache/spark/spark-2.3.0/spark-2.3.0-bin-hadoop2.7.tgz
3.解压
tar -zxvf spark-2.3.0-bin-hadoop2.7.tgz
4.复制到/usr/local文件夹下面
mv spark-2.3.0-bin-hadoop2.7 /usr/local/spark
5.修改配置文件conf/spark-env.sh
cd /usr/local/spark/conf
nano spark-env.sh.template
写入
#java安装目录
JAVA_HOME=/usr/lib/jvm/java-8-oracle
#scala安装目录
SCALA_HOME=/usr/local/scala
#hadoop 配置文件的目录(HA必须指定)
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop/
##Spark集群配置参数##
#配置master的端口号
SPARK_MASTER_PORT=7077
#配置master的web端口号
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1000m
SPARK_WORKER_PORT=7078
SPARK_WORKER_WEBUI_PORT=8081
SPARK_WORKER_INSTANCES=1
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=
ZOOKEEPER -Dspark.deploy.zookeeper.url=
master01:2181,master02:2181,slaver01:2181,slaver02:2181 -Dspark.deploy.zookeeper.dir=/spark"
6.修改配置文件conf/slaves.template(最后保存为slaves)
slaver01
slaver02
7.复制文件夹到各个系统,并配置spark的环境变量
scp -r spark root@master02:/usr/local/
scp -r spark root@slaver01:/usr/local/
scp -r spark root@slaver02:/usr/local/
8.启动集群
启动顺序:Zookeeper -> Hadoop -> Spark
SparkMaster节点执行:
【SPARK_HOME】/sbin/start-all.sh
- 1
- 2
SparkMaster热备节点执行:
【SPARK_HOME】/sbin/start-master.sh
9.检查
在web浏览器栏输入:http://【SPARK_MASTER】:8080/
参考:https://blog.csdn.net/u014726937/article/details/52093049