大数据 Apache Flink的HA集群搭建

官方地址连接

• 独立集群的JobManager高可用性的一般思想是，在任何时候都有一个单一的领先JobManager和多个备用JobManager来接管领导层，以防领导者失败。这就保证了没有单点故障，只要有一个待命的作业经理担任领导，程序就可以取得进展。待机和主JobManager实例之间没有明确的区别。每个JobManager都可以担任master或standby角色。

搭建过程

先决条件（略）

• 安装JDK
• 安装HADOOP HDFS-HA
• 安装Zookeeper

Flink环境构建

• 配置HADOOP_CLASSPATH

[root@CentOSX ~]# vi .bashrc
HADOOP_HOME=/usr/hadoop-2.9.2
JAVA_HOME=/usr/java/latest
PATH=$PATH:$/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
CLASSPATH=.
export JAVA_HOME
export PATH
export CLASSPATH
export HADOOP_HOME
HADOOP_CLASSPATH=`hadoop classpath`
export HADOOP_CLASSPATH
[root@CentOSX ~]# source .bashrc
[root@CentOSX ~]# echo $HADOOP_CLASSPATH

• 上传Flink,配置Flink

[root@CentOSX ~]# tar -zxf flink-1.8.2-bin-scala_2.11.tgz -C /usr/
[root@CentOSA ~]# cd /usr/flink-1.8.2
[root@CentOSA flink-1.8.1]# vi conf/flink-conf.yaml

# Common
#==============================================================================
taskmanager.numberOfTaskSlots: 4
parallelism.default: 3
#==============================================================================
# High Availability
#==============================================================================
 high-availability: zookeeper
 high-availability.storageDir: hdfs:///flink/ha/
 high-availability.zookeeper.quorum: CentOSA:2181,CentOSB:2181,CentOSC:2181
 high-availability.zookeeper.path.root: /flink
 high-availability.cluster-id: /default_ns
#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================
 state.backend: rocksdb
 state.checkpoints.dir: hdfs:///flink-checkpoints
 state.savepoints.dir: hdfs:///flink-savepoints
 state.backend.incremental: true

• 配置masters文件

[root@CentOSX flink-1.8.2]# vi conf/masters
CentOSA:8081
CentOSB:8081
CentOSC:8081

• 配置slaves文件

[root@CentOSA flink-1.8.2]# vi conf/slaves
CentOSA
CentOSB
CentOSC

启动Flink集群

[root@CentOSA flink-1.8.2]# ./bin/start-cluster.sh
Starting HA cluster with 3 masters.
Starting standalonesession daemon on host CentOSA.
Starting standalonesession daemon on host CentOSB.
Starting standalonesession daemon on host CentOSC.
Starting taskexecutor daemon on host CentOSA.
Starting taskexecutor daemon on host CentOSB.
Starting taskexecutor daemon on host CentOSC.

• 等集群启动完成后，查看JobManager任务的日志，在lead主机中可以看到：

http://xxx:8081 was granted leadership with leaderSessionID=f5338c3f-c3e5-4600-a07c-566e38bc0ff4

测试HA

• 登陆获取leadership的节点，然后执行以下指令

[root@CentOSB flink-1.8.2]# ./bin/jobmanager.sh stop

• 查看其它节点，按照上诉的测试方式，可以查找leadership日志输出的节点，该节点就是master节点。