Flink代码阅读之部署方式

最新推荐文章于 2024-08-16 15:14:35 发布

define_us

最新推荐文章于 2024-08-16 15:14:35 发布

阅读量119

点赞数

分类专栏： java

本文链接：https://blog.csdn.net/define_us/article/details/113761903

版权

java 专栏收录该内容

221 篇文章 0 订阅

订阅专栏

![在这里插入图片描述](https://img-blog.csdnimg.cn/202102082012468.png

客户端提交任务的方式主要有以下几种

命令行
REST接口
SQL
python
scala

JobManager有三种提交任务的模式

Application Mode: runs the cluster exclusively for one application. The job’s main method (or client) gets executed on the JobManager. Calling execute/executeAsync multiple times in an application is supported.
1.11版本以后才会有的。本文暂不涉及。
Per-Job Mode: runs the cluster exclusively for one job. The job’s main method (or client) runs only prior to the cluster creation.
每一个任务都会启动一个集群。
Session Mode: one JobManager instance manages multiple jobs sharing the same cluster of TaskManagers
集群只有一个JobManager。各个job共享TaskManager

部署方式

无论如何部署，都是大概这四种进程。注意图片最下面master/yarn只是代表部署在yarn的AM节点。
在这里插入图片描述
+

对于JM，有三个组成部分

The JobManager has a number of responsibilities related to coordinating the distributed execution of Flink Applications: it decides when to schedule the next task (or set of tasks), reacts to finished tasks or execution failures, coordinates checkpoints, and coordinates recovery on failures, among others. This process consists of three different components:

ResourceManager 注意这个RM和yarn的rm两件事情。

The ResourceManager is responsible for resource de-/allocation and provisioning in a Flink cluster — it manages task slots, which are the unit of resource scheduling in a Flink cluster (see TaskManagers). Flink implements multiple ResourceManagers for different environments and resource providers such as YARN, Mesos, Kubernetes and standalone deployments. In a standalone setup, the ResourceManager can only distribute the slots of available TaskManagers and cannot start new TaskManagers on its own.

Dispatcher

The Dispatcher provides a REST interface to submit Flink applications for execution and starts a new JobMaster for each submitted job. It also runs the Flink WebUI to provide information about job executions.

JobMaster

A JobMaster is responsible for managing the execution of a single JobGraph. Multiple jobs can run simultaneously in a Flink cluster, each having its own JobMaster.

Resorce Provider的不同，部署方式有如下四种
Standalone：最基本模式。
Kubernetes
YARN
Mesos