Flink1.17.0 Standalone高可用集群搭建

狂奔的蜗牛& 不可知

已于 2023-09-08 11:58:10 修改

阅读量320

点赞数

文章标签： flink

于 2023-09-07 11:57:08 首次发布

本文链接：https://blog.csdn.net/qq_28604399/article/details/132734755

版权

前置条件： jdk1.8，节点间免密，关闭防火墙，关闭selinux，设置主机域名并配置到hosts文件中，修改Linux部署用户的最大进程数，修改Linux部署用户的打开最大文件数，并搭建好zookeeper集群、hadoop集群。

修改配置

1.flink-conf.yaml

jobmanager.rpc.address: master JobManager地址，高可用时，zookeeper动态选举设置
jobmanager.rpc.port: 6123     JobManagerRPC通信端口
jobmanager.heap.size: 1024m   JobManager所能使用的堆内存大小
taskmanager.heap.size: 1024m  TaskManager所能使用的堆内存大小
taskmanager.numberOfTaskSlots: 2 TaskManager管理的TaskSlot个数，依据当前物理机的核心数来配置，一般预留出一部分核心（25%）给系统及其他进程使用，一个slot对应一个core。如果core支持超线程，那么slot个数*2
rest.bind-address: 0.0.0.0
rest.port: 8081 指定WebUI的访问端口

#==============================================================================
# High Availability
#==============================================================================

# The high-availability mode. Possible options are 'NONE' or 'zookeeper'.
#
high-availability.type: zookeeper

# The path where metadata for master recovery is persisted. While ZooKeeper stores
# the small ground truth for checkpoint and leader election, this location stores
# the larger objects, like persisted dataflow graphs.
# 
# Must be a durable file system that is accessible from all nodes
# (like HDFS, S3, Ceph, nfs, ...) 
#
high-availability.storageDir: hdfs://hacluster/flink/ha/

# The list of ZooKeeper quorum peers that coordinate the high-availability
# setup. This must be a list of the form:
# "host1:clientPort,host2:clientPort,..." (default clientPort: 2181)
#
high-availability.zookeeper.quorum: master:2181,slave01:2181,slave02:2181


# The address under which the web-based HistoryServer listens.
historyserver.web.address: 0.0.0.0

# The port under which the web-based HistoryServer listens.
historyserver.web.port: 8082

# Comma separated list of directories to monitor for completed jobs.
historyserver.archive.fs.dir: hdfs://hacluster/completed-jobs/

2.masters