spark ha安装

最新推荐文章于 2023-01-12 14:29:10 发布

xiecj_2006

最新推荐文章于 2023-01-12 14:29:10 发布

阅读量231

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/xiecj_2006/article/details/103643098

版权

大数据专栏收录该内容

4 篇文章 0 订阅

订阅专栏

### 下载spark
```
wget https://www.apache.org/dyn/closer.lua/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz
```
### 解压缩
```
tar zxvf spark-2.4.3-bin-hadoop2.7.tgz -C /opt/module/
```
### 拷贝初始文件
```
cp spark-env.sh.template spark-env.sh
cp spark-defaults.conf.template spark-defaults.conf
cp slaves.template slaves
```
### 修改spark-env.sh
```
grep -vE '^#|^$' spark-env.sh
export SPARK_DIST_CLASSPATH=$(/opt/module/hadoop-3.2.0/bin/hadoop classpath)
JAVA_HOME=/opt/module/jdk1.8.0_211
SCALA_HOME=/opt/module/scala-2.13.0
HADOOP_CONF_DIR=/opt/module/hadoop-3.2.0/etc/hadoop
HADOOP_HOME=/opt/module/hadoop-3.2.0
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop11:2181,hadoop12:2181,hadoop13:2181 -Dspark.deploy.zookeeper.dir=/spark"
```
### 设置spark-defaults.conf
```
grep -vE '^#|^$' conf/spark-defaults.conf
spark.master spark://hadoop11:7077,hadoop12:7077
```
### 修改slaves
```
grep -vE '^#|^$' slaves
hadoop13
hadoop14
```
### 增加环境变量
```
## SPARK_HOME
export SPARK_HOME=/opt/module/spark-2.4.3-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin
```
### 同步数据
```
xsync.sh spark-2.4.3-bin-hadoop2.7/
```
### 启动spark
```
./sbin/start-all.sh
```
### 检查
```
xcall.sh jps
================== hadoop11 jps==================
3056 QuorumPeerMain
3745 DFSZKFailoverController
3539 JournalNode
10691 Master
10875 Jps
3294 NameNode
3918 NodeManager
6990 HMaster
================== hadoop12 jps==================
7796 Master
2629 NameNode
2935 DFSZKFailoverController
3111 NodeManager
2793 JournalNode
4073 HMaster
2492 QuorumPeerMain
7884 Jps
2703 DataNode
================== hadoop13 jps==================
2693 JournalNode
6631 Worker
2457 QuorumPeerMain
3066 ResourceManager
6747 Jps
2588 DataNode

[hadoop@hadoop11 spark-2.4.3-bin-hadoop2.7]$ ./bin/spark-shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019-06-13 17:09:04,701 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://hadoop11:4040
Spark context available as 'sc' (master = spark://hadoop11:7077,hadoop12:7077, app id = app-20190613170930-0001).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.3
/_/