Flink v1.6.1 《Yarn 模式高可用分布式集群部署》

最新推荐文章于 2024-07-24 11:58:44 发布

Mathieu66

最新推荐文章于 2024-07-24 11:58:44 发布

阅读量547

点赞数

分类专栏： Apache Flink 文章标签： Flink yarn HA 高可用分布式集群部署

本文链接：https://blog.csdn.net/Mathieu66/article/details/83621305

版权

Apache Flink 专栏收录该内容

19 篇文章 1 订阅

订阅专栏

Flink高可用集群分为Standalone HA 和 YARN Cluster HA，这里只介绍 YARN Cluster HA。

1. Flink下载

安装包下载地址：https://archive.apache.org/dist/flink/flink-1.6.1/flink-1.6.1-bin-hadoop26-scala_2.11.tgz

选择与Hadoop版本对应的安装包

2. 环境准备

2.1 软件要求

Java 1.8.x或更高版本
ssh(运行Flink 脚本管理远程组件必须运行sshd)

2.2 集群部署规划

主机名	Master	Worker	Zookeeper
master			zookeeper
node1		worker
node2		worker
node3		worker	zookeeper
node4		worker	zookeeper

Yarn HA集群只运行一个JobManager ，具体是哪个节点由zookeeper选举决定，所以并不清楚会在哪台上。

3. 解压安装

3.1 解压安装包

tar -zxvf flink-1.6.1-bin-hadoop26-scala_2.11.tgz -C /opt/

3.2 修改配置文件

[root@master conf]# cd /opt/flink-1.6.1/conf/
[root@master conf]# ls
flink-conf.yaml           log4j.properties               logback.xml       slaves
log4j-cli.properties      log4j-yarn-session.properties  logback-yarn.xml  sql-client-defaults.yaml
log4j-console.properties  logback-console.xml            masters           zoo.cfg
[root@master conf]#

Yarn HA 模式需要配置三个文件：flink-conf.yaml，slaves，zoo.cfg。（master选举产生，不用配置）

flink-conf.yaml

high-availability: zookeeper
high-availability.zookeeper.quorum: master:2181,node3:2181,node4:2181
high-availability.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.path.root: /flink
yarn.application-attempts: 10

slaves

node1
node2
node3
node4

zoo.cfg

# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial  synchronization phase can take
initLimit=10

# The number of ticks that can pass between  sending a request and getting an acknowledgement
syncLimit=5

# The directory where the snapshot is stored.
# dataDir=/tmp/zookeeper

# The port at which the clients will connect
clientPort=2181

# ZooKeeper quorum peers
server.1=master:2888:3888
server.2=node3:2888:3888
server.3=node4:2888:3888

此外，还需要在yarn-site.xml中配置 application masters尝试的最大次数（默认2）。

<property>
  <name>yarn.resourcemanager.am.max-attempts</name>
  <value>4</value>
  <description>
    The maximum number of application master execution attempts.
  </description>
</property>

3.3 发送到其他节点

[root@master conf]# cd ../../
[root@master opt]# scp -r flink-1.6.1 root@node1:/opt/

每个节点都发一次

3.4 配置环境变量

[root@master opt]# echo 'export HADOOP_CONF_DIR=/etc/hadoop/conf' >> /etc/profile
[root@master opt]# echo 'export FLINK_HOME=/opt/flink-1.6.1' >> /etc/profile
[root@master opt]# echo 'export PATH=$PATH:$FLINK_HOME/bin' >> /etc/profile
[root@master opt]# source /etc/profile