Flink 1.7.2源码分析

最新推荐文章于 2024-05-07 16:46:12 发布

置顶 thinktothings

最新推荐文章于 2024-05-07 16:46:12 发布

阅读量1.1k

点赞数

分类专栏： Flink 文章标签： Flink 1.7.2 源码分析 Flink学习笔记

本文链接：https://blog.csdn.net/thinktothings/article/details/88262633

版权

Flink 专栏收录该内容

16 篇文章 0 订阅

订阅专栏

技术交流

微信:thinktothings
微博:https://weibo.com/thinktothings
Flink版本为1.7.2
本站持续更新中…2019-03-13…

源码

https://github.com/opensourceteams/flink-maven-scala

中文文档

Flink 1.7中文文档(官网英文翻译过来) 在线版: http://flink.apachecn.org/docs/1.7-SNAPSHOT/#/README
Flink 1.7中文文档(官网英文翻译过来) PDF版: https://github.com/opensourceteams/flink-maven-scala/blob/master/doc/flink-1.7-中文文档(官网英文翻译过来).pdf

Flink 本地运行交互Shell

start-scala-shell.sh local
参数说明: [local | remote | yarn]

benv.fromElements(1,2,3).map(i => i * i ).print

输出结果

1
4
9

运行 jar 到 Flink 集群

 flink run -c  com.opensourceteams.module.bigdata.flink.example.stream.worldcount.nc.SocketWindowWordCount    ./flink-maven-scala-2-0.0.1.jar

创建flink java 项目

mvn archetype:generate                               \
      -DarchetypeGroupId=org.apache.flink              \
      -DarchetypeArtifactId=flink-quickstart-java     \
      -DarchetypeVersion=1.7.1
      -DgroupId=com.opensourceteams \
      -DartifactId=flink-maven-java \
      -Dversion=0.0.1 \
      -Dpackage=com.opensourceteams.module.bigdata.flink  \
      -DinteractiveMode=false

创建flink scala项目

mvn archetype:generate                               \
      -DarchetypeGroupId=org.apache.flink              \
      -DarchetypeArtifactId=flink-quickstart-scala     \
      -DarchetypeVersion=1.7.1
      -DgroupId=com.opensourceteams \
      -DartifactId=flink-maven-scala-2 \
      -Dversion=0.0.1 \
      -Dpackage=com.opensourceteams.module.bigdata.flink  \
      -DinteractiveMode=false

查看jar中文件列表

jar tvf test.jar

maven 运行某个类

mvn exec:java -Dexec.mainClass=wikiedits.WikipediaAnalysis

执行计划图

用Firefox 打开，显示的比较全(别浏览器有显示不全的现象)
地址:https://flink.apache.org/visualizer

      //执行计划
      //println(env.getExecutionPlan)
      //StreamGraph
     //println(env.getStreamGraph.getStreamingPlanAsJSON)

Execute Plan

{"nodes":[{"id":1,"type":"Source: Socket Stream","pact":"Data Source","contents":"Source: Socket Stream","parallelism":1},{"id":2,"type":"Flat Map","pact":"Operator","contents":"Flat Map","parallelism":1,"predecessors":[{"id":1,"ship_strategy":"FORWARD","side":"second"}]},{"id":3,"type":"Map","pact":"Operator","contents":"Map","parallelism":1,"predecessors":[{"id":2,"ship_strategy":"FORWARD","side":"second"}]},{"id":5,"type":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","pact":"Operator","contents":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","parallelism":1,"predecessors":[{"id":3,"ship_strategy":"HASH","side":"second"}]},{"id":6,"type":"Sink: Print to Std. Out","pact":"Data Sink","contents":"Sink: Print to Std. Out","parallelism":1,"predecessors":[{"id":5,"ship_strategy":"FORWARD","side":"second"}]}]}

StreamGraph Plan

{"nodes":[{"id":1,"type":"Source: Socket Stream","pact":"Data Source","contents":"Source: Socket Stream","parallelism":1},{"id":2,"type":"Flat Map","pact":"Operator","contents":"Flat Map","parallelism":1,"predecessors":[{"id":1,"ship_strategy":"FORWARD","side":"second"}]},{"id":3,"type":"Map","pact":"Operator","contents":"Map","parallelism":1,"predecessors":[{"id":2,"ship_strategy":"FORWARD","side":"second"}]},{"id":5,"type":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","pact":"Operator","contents":"Window(TumblingProcessingTimeWindows(3000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction)","parallelism":1,"predecessors":[{"id":3,"ship_strategy":"HASH","side":"second"}]},{"id":6,"type":"Sink: Print to Std. Out","pact":"Data Sink","contents":"Sink: Print to Std. Out","parallelism":1,"predecessors":[{"id":5,"ship_strategy":"FORWARD","side":"second"}]}]}

Flink 环境，配置

Flink 源码debug方法: https://github.com/opensourceteams/flink-maven-scala/blob/master/md/flink-debug.md
Flink 名词术语 : https://github.com/opensourceteams/flink-maven-scala/blob/master/md/flink-concept.md
Flink 源码编译: https://github.com/opensourceteams/flink-maven-scala/blob/master/md/flink-build.md

example

scala 版Flink WordCount单词统计 : https://github.com/opensourceteams/flink-maven-scala/blob/master/md/wordCount-scala.md

wordCount Dataset批处理

start-scala-shell.sh local
参数说明:[local | remote | yarn]

benv.fromElements("a b a c").flatMap(x => x.split(" ")).map((_,1)).groupBy(0).sum(1).print

输出结果

  (a,2)
  (b,1)
  (c,1)

Flink 1.7.2 DataStream operator 示例 : https://github.com/opensourceteams/flink-maven-scala/blob/master/md/datastream/flink-datastream-operator.md
Flink1.7.2 Dataset transformation示例: https://github.com/opensourceteams/flink-maven-scala/blob/master/md/dataset/flink-dataset-transformation.md

关注

0
点赞
踩
5

收藏

觉得还不错? 一键收藏
0
评论
Flink 1.7.2源码分析

技术交流微信:thinktothings微博:https://weibo.com/thinktothingsFlink版本为1.7.2本站持续更新中…2019-03-06…源码https://github.com/opensourceteams/flink-maven-scala查看jar中文件列表jar tvf test.jar 执行计划图地址:https://...
复制链接

扫一扫