9.4 Spark安装
9.4.1 下载spark
去官网https://spark.apache.org/downloads.html查找合适的版本,这里选用【spark-3.1.2-bin-hadoop2.7.tgz】
进行创建目录、下载、解压、改名四连操作
# cd /opt
# wget https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop2.7.tgz
# tar -zxvf spark-3.1.2-bin-hadoop2.7.tgz
# mv spark-3.1.2-bin-hadoop2.7 spark
9.4.2 配置环境变量
# vi /etc/profile
新增
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin
拷贝到其他虚拟机
# scp /etc/profile root@hadoop02:/etc
# scp /etc/profile root@hadoop03:/etc
# scp /etc/profile root@hadoop04:/etc
# scp /etc/profile root@hadoop05:/etc
在各机器分别重启环境变量
# source /etc/profile
9.4.3 修改spark的配置文件
# cd /opt/spark/conf/
# cp spark-env.sh.template spark-env.sh # 拷贝一份配置文件
# vi spark-env.sh
新增
export JAVA_HOME=/opt/jdk1.8.0_141
export HADOOP_HOME=/opt/hadoop-2.10.1
export HADOOP_CONF_DIR=/opt/hadoop-2.10.1/etc/hadoop
# export SPARK_MASTER_IP=hadoop01 # 集群中Master地址是不固定的 所以必须把这行注释掉
export SPARK_MASTER_PORT=7077
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop01:2181,hadoop02:2181,hadoop03:2181,hadoop04:2181,hadoop05:2181 -Dspark.deploy.zookeeper.dir=/spark"
如果需要使用浏览器查看日志则需要开启历史日志服务:
# cp spark-defaults.conf.template spark-defaults.conf
# vi spark-defaults.conf
// 加入:
spark.master spark://hadoop01:7077
# spark.eventLog.enabled true
# spark.eventLog.dir hdfs://hadoop01:8021/directory
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 5g
spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
![image-20210819132422915](https://tva1.sinaimg.cn/large/008i3skNgy1gtm15etvxmj611u0n8wjo02.jpg)
重命名并修改slaves.template文件
# mv workers.template workers
# vi workers
加上
hadoop02
hadoop03
hadoop04
hadoop05
拷贝到其他机器
# cd /opt
# scp -r spark hadoop02:$PWD
# scp -r spark hadoop03:$PWD
# scp -r spark hadoop04:$PWD
# scp -r spark hadoop05:$PWD
9.4.4 启动Spark
启动zk,查看zk是否已经启动
# /opt/zookeeper/bin/zkServer.sh status
启动spark
在hadoop01启动
# sh /opt/spark/sbin/start-all.sh
![image-20210819111541887](https://tva1.sinaimg.cn/large/008i3skNgy1gtlxfedrlsj61qs05on0i02.jpg)
在hadoop02启动master
# sh /opt/spark/sbin/start-master.sh
![image-20210819112637005](https://tva1.sinaimg.cn/large/008i3skNgy1gtlxqqs0jsj61kk02wq3o02.jpg)
9.4.5端口查看
![image-20210819112737193](https://tva1.sinaimg.cn/large/008i3skNgy1gtlxrtkumtj61ue0o210z02.jpg)
在服务器把hadoop01的8080端口转给服务器的8080端口,访问 【服务器ip:8080】
![image-20210819112340575](https://tva1.sinaimg.cn/large/008i3skNgy1gtlxnp90yej61c00u0tdn02.jpg)
可以看到hadoop01的spark为active状态
在服务器把hadoop02的8080端口转给服务器的8080端口,访问 【服务器ip:8081】
![image-20210819112611207](https://tva1.sinaimg.cn/large/008i3skNgy1gtlxqb5lamj61c00u0adm02.jpg)
hadoop02的spark为standby状态
测试停用hadoop01的master
# jps
![image-20210819112903322](https://tva1.sinaimg.cn/large/008i3skNgy1gtlxta91laj60k006eq3b02.jpg)
# kill -9 38533
刷新hadoop2的spark
![image-20210819113039943](https://tva1.sinaimg.cn/large/008i3skNgy1gtlxuz3zi6j61c00u00xo02.jpg)
发现这时hadoop02的spark已经接替了hadoop01进行工作
9.4.6 测试spark
启动spark shell,方便用户进行交互式编程,用户可以在该命令行下用scala编写spark程序。
# hdfs dfs -mkdir -p /spark/input
# cd /opt
# vi test
// 输入测试文本
I LOVE TONGJI
I AM CHINESE
I LOVE CHINA
# hdfs dfs -put test /spark/input