目录
3、用scp把配置好的spark文件夹传到slave1和slave2上
1、准备好安装包到master上并配置环境变量
vim /etc/profile
#JAVA
export JAVA_HOME=/opt/bigdata/jdk
export PATH=${JAVA_HOME}/bin:$PATH
export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
export JRE_HOME=${JAVA_HOME}/jre
#hbase
export HBASE_HOME=/opt/bigdata/hbase
export PATH=${HBASE_HOME}/bin:$PATH
#SCALA
export SCALA_HOME=/opt/bigdata/scala
export PATH=${SCALA_HOME}/bin:$PATH
#SPARK
export SPARK_HOME=/opt/bigdata/spark
export PATH=${SPARK_HOME}/bin:$PATH
#SBT
export SBT_HOME=/opt/bigdata/sbt
export PATH=${SBT_HOME}/bin:$PATH
#hadoop
export HADOOP_HOME=/opt/bigdata/hadoop
export PATH=${HADOOP_HOME}/bin:$PATH
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib"
2、对spark的配置文件进行配置
spark的配置文件都在/opt/bigdata/spark/conf目录下
2.1 spark-env.sh
在这里指定了hadoop的配置文件路径后不用再把hadoop的配置文件拷到conf的目录下
[root@master conf]# vim spark-env.sh
export JAVA_HOME=/opt/bigdata/jdk
export SCALA_HOME=/opt/bigdata/scala
export SPARK_MASTER_IP=10.103.104.101
export SPARK_DIST_CLASSPATH=$(/opt/bigdata/hadoop/bin/hadoop classpath)
export HADOOP_CONF_DIR=/opt/bigdata/hadoop/etc/hadoop/
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://westgis101:9000/sparkJobhistory"
2.2 slaves
[root@master conf]# vim slaves
slave1
slave2
👆为两台slave的机器名
3、用scp把配置好的spark文件夹传到slave1和slave2上
scp -r /opt/bigdata/spark slave1:/opt/bigdata/
scp -r /opt/bigdata/spark slave2:/opt/bigdata/
4、启动hadoop和spark
[root@slave1 bigdata]# ./hadoop/sbin/start-all.sh
[root@slave1 bigdata]# ./spark/sbin/start-all.sh
5、jps查看进程运行情况
[root@slave1 bigdata]# jps
8067 Worker
7927 DataNode
8559 Jps
[root@slave1 bigdata]# ssh master
Last login: Mon Nov 4 11:01:06 2019 from 192.168.19.1
[root@master ~]# jps
8180 NameNode
8470 SecondaryNameNode
9688 Jps
8589 SparkSubmit
8765 Master