spark-master 高可用测试

最新推荐文章于 2023-11-27 23:47:23 发布

baya2295

最新推荐文章于 2023-11-27 23:47:23 发布

阅读量182

点赞数

文章标签：大数据

原文链接：http://blog.51cto.com/michaelkang/2421714

版权

本文详细记录了Spark高可用集群的搭建与测试过程，包括三台master节点与三台slave节点的配置，以及通过Zookeeper进行leader选举的机制。文章展示了如何启动与关闭master节点，以及在leader节点失效后，集群如何自动选举出新的leader。

摘要由CSDN通过智能技术生成

spark-master 高可用测试：

集群介绍

[spark-m]
172.20.101.157 sparkname=master01 
172.20.101.164 sparkname=master02
172.20.101.165 sparkname=master03 

[spark-s]
172.20.101.160 sparkname=slave01 
172.20.101.166 sparkname=slave02
172.20.101.167 sparkname=slave03 

[zookeeper]
172.20.101.157 myid=1 
172.20.101.164 myid=3 
172.20.101.165 myid=2

启动第1台master：

[root@kubm-03 ~]# tailf /var/log/spark/spark-root-org.apache.spark.deploy.master.Master-1-kubm-03.out 
.......
19/07/18 11:11:59 INFO ZooKeeperLeaderElectionAgent: We have gained leadership
19/07/18 11:11:59 INFO Master: I have been elected leader! New state: ALIVE

发现第1台未发现其它节点，状态选举为 leader

启动第2台master：

[root@kubm-02 ~]# sh /usr/local/spark-2.4.3-bin-hadoop2.7/sbin/start-master.sh 
.......
19/07/18 11:17:52 INFO ConnectionStateManager: State change: CONNECTED

状态选举为 CONNECTED

启动第2台master：

[root@kubm-01 ~]# sh /usr/local/spark-2.4.3-bin-hadoop2.7/sbin/start-master.sh 
.。。。。
19/07/18 11:15:09 INFO ConnectionStateManager: State change: CONNECTED

状态选举为 CONNECTED

关闭 leader 节点服务

[root@kubm-03 ~]# sh /usr/local/spark-2.4.3-bin-hadoop2.7/sbin/stop-master.sh

略等待，kubm-01节点选举为新的 leader

[root@kubm-01 ~]# 
19/07/18 11:19:33 INFO ZooKeeperLeaderElectionAgent: We have gained leadership
19/07/18 11:19:33 INFO Master: I have been elected leader! New state: ALIVE