架构和处理模型

翻译 2015年11月20日 16:55:57


The Processes

When the Flink system is started, it bring up the JobManager and one or more TaskManagers. The JobManager is the coordinator of the Flink system, while the TaskManagers are the workers that execute parts of the parallel programs. When starting the system in local mode, a single JobManager and TaskManager are brought up within the same JVM.

When a program is submitted, a client is created that performs the pre-processing and turns the program into the parallel data flow form that is executed by the JobManager and TaskManagers. The figure below illustrates the different actors in the system and their interactions.

Flink Process Model

Component Stack

As a software stack, Flink is a layered system. The different layers of the stack build on top of each other and raise the abstraction level of the program representations they accept:

  • The runtime layer receives a program in the form of a JobGraph. A JobGraph is a generic parallel data flow with arbitrary tasks that consume and produce data streams.

  • Both the DataStream API and the DataSet API generate JobGraphs through separate compilation processes. The DataSet API uses an optimizer to determine the optimal plan for the program, while the DataStream API uses a stream builder.

  • The JobGraph is executed according to a variety of deployment options available in Flink (e.g., local, remote, YARN, etc)

  • Libraries and APIs that are bundled with Flink generate DataSet or DataStream API programs. These are Table for queries on logical tables, FlinkML for Machine Learning, and Gelly for graph processing.

You can click on the components in the figure to learn more.

Stack

Graph API: GellyFlink MLTableDataSet API (Java/Scala)DataStream API (Java/Scala)Flink RuntimeLocalRemoteEmbeddedYARNTez

Projects and Dependencies

The Flink system code is divided into multiple sub-projects. The goal is to reduce the number of dependencies that a project implementing a Flink progam needs, as well as to faciltate easier testing of smaller sub-modules.

The individual projects and their dependencies are shown in the figure below.

The Flink sub-projects and their dependencies

In addition to the projects listed in the figure above, Flink currently contains the following sub-projects:

  • flink-dist: The distribution project. It defines how to assemble the compiled code, scripts, and other resources into the final folder structure that is ready to use.

  • flink-staging: A series of projects that are in an early version. Currently contains among other things projects for YARN support, JDBC data sources and sinks, hadoop compatibility, graph specific operators, and HBase connectors.

  • flink-quickstart: Scripts, maven archetypes, and example programs for the quickstarts and tutorials.

  • flink-contrib: Useful tools contributed by users. The code is maintained mainly by external contributors. The requirements for code being accepted into flink-contrib are lower compared to the rest of the code.


bs模型与cs模型

bs模式 客户端通过浏览器,浏览web服务器上的网页,这样的模型叫bs模型,b指客户端browser,s指服务端server。在客户端和浏览器端之间走的报文是http协议(即超文本传输协议)   ...
  • baoxiaofeicsdn
  • baoxiaofeicsdn
  • 2015年09月09日 13:57
  • 817

【设计模式 2】设计模式的来源,及其相混淆的模型、框架、架构区别

导读:周日,又给徒弟验收项目了。以往常的习惯一样,每次验收,我都陪着徒弟一起总结学习过程中的问题,然后互相交流。这次,一个徒弟验收的是设计模式,一个徒弟验收的是UML,本篇博客,就先谢谢设计模式的东西...
  • u013034889
  • u013034889
  • 2015年12月23日 20:33
  • 970

统一架构模型(UAM)

UAM(Unified Architecture Model),软件设计研发的统一架构技术模型,是对软件研发设计中需求分析、架构设计、框架设计、功能设计、功能研发、后台数据模型设计、数据管理、门户平台...
  • ShanYeRen2016
  • ShanYeRen2016
  • 2016年05月05日 21:46
  • 361

移动App框架模式了解【待看】

移动App架构设计 本文主要总结了几种常用的架构模式, 基本是层层递进的转载请注名出处 http://blog.csdn.net/uxyheaven, 良好的排版在https://github.co...
  • adparking
  • adparking
  • 2016年12月06日 11:07
  • 569

持续交付的架构成熟度模型

随着云和容器技术的发展,大家对DevOps和CI/CD的重要性有了更深入的认识。今天我们就讨论一下架构设计如何更好的支持CI/CD。 什么是持续集成,交付和部署(CI/CD) Martin Fow...
  • github_39335046
  • github_39335046
  • 2017年06月29日 10:54
  • 333

B/S模型基本架构

B/S是一种常见的客户端与服务器的交互模型,具有很多优点,这些优点使得它越来越流行。B/S模型具有以下优点:   客户端:客户端使用主要是浏览器,浏览器为用户提供了统一的操作平台,即对于不同的服务器...
  • xiaofangzi11
  • xiaofangzi11
  • 2016年11月22日 13:41
  • 1363

架构的分析模型

架构的分析模型 一、 讨论的背景 现代电子游戏,基本上都会使用一定的网络功能。从验证正版,到多人交互等等,都需要架设一些专用的服务器,以及编写在服务器上的程序。因此,游戏服务器端软件的...
  • zhaoguanghui2012
  • zhaoguanghui2012
  • 2016年08月12日 18:15
  • 1076

我的架构经验小结(一)-- 常用的架构模型

 经过这几年的积累,在系统架构方面逐渐积累了一些自己的经验,到今天有必要对这些经验作个小结。在我的架构思维中,主要可以归类为三种架构模型:3/N层架构、“框架+插件”架构、地域分布式架构。一.三种架构...
  • wishfly
  • wishfly
  • 2007年12月02日 16:23
  • 645

The Onion Architecture(洋葱架构) part3

In my previous installments, I described what has become my approach to defining the architecture fo...
  • zhang_qxian
  • zhang_qxian
  • 2018年01月19日 15:21
  • 50

icloud之部署架构

Deploying the Schema 当您完成您的架构和测试在开发环境中您的应用程序,你准备好部署模式来生产。部署促进模式向生产环境中,但它不会在开发环境中,以生产环境中复制记录。因此,在部署之...
  • zzzzzdddddxxxxx
  • zzzzzdddddxxxxx
  • 2015年09月16日 17:41
  • 440
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:架构和处理模型
举报原因:
原因补充:

(最多只允许输入30个字)