1.上传spark-2.2.2-bin-hadoop2.6.tgz到/opt目录,并解压到/usr/local
tar -zxf /opt/spark-2.2.2-bin-hadoop2.6.tgz -C /usr/local/
2.在所有节点配置Spark环境变量
vi /etc/profile
在文件尾加入:
export SPARK_HOME=/usr/local/spark-2.2.2-bin-hadoop2.6
export PATH=$PATH:$SPARK_HOME/bin
执行source /etc/profile使命令生效
3.进入/usr/local/spark-2.2.2-bin-hadoop2.6/conf
复制slaves.template:cp slaves.template slaves
修改slaves:
vi slaves
删去localhost,添加内容:
slave1
slave2
slave3
4.修改spark-defaults.conf
cp spark-defaults.conf.template spark-defaults.conf
vi spark-defaults.conf
添加:
spark.master spark://master:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://master:8020/spark-logs
spark.history.fs.logDirectory hdfs://master:8020/spark-logs
5.修改spark-env.sh
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
添加:
JAVA_HOME=/usr/java/jdk1.8.0_151
HADOOP_CONF_DIR=/usr/local/hadoop-2.6.4/etc/hadoop
SPARK_MASTER_IP=master
SPARK_MASTER_PORT=7077
SPARK_WORKER_MEMORY=512m
SPARK_WORKER_CORES=1
SPARK_EXECUTOR_MEMORY=512m
SPARK_EXECUTOR_CORES=1
SPARK_WORKER_INSTANCES=1
6.在HDFS中新建目录:(先打开Hadoop集群,且需到hadoop的bin目录下输入hadoop dfsadmin -safemode leave解除安全模式)
hdfs dfs -mkdir /spark-logs
7.将Spark安装包分发到其他节点
scp -r /usr/local/spark-2.2.2-bin-hadoop2.6/ slave1:/usr/local/
scp -r /usr/local/spark-2.2.2-bin-hadoop2.6/ slave2:/usr/local/
scp -r /usr/local/spark-2.2.2-bin-hadoop2.6/ slave3:/usr/local/
8.启动spark
进入
/usr/local/spark-2.2.2-bin-hadoop2.6/sbin
执行
./start-all.sh
./start-history-server.sh hdfs://master:8020/spark-logs
(关闭则将start改为stop)
9.查看客户端
http://master:8080
http://master:18080/
10.进入Spark命令页面
spark-shell(:q退出)