Flink 1.12.2 源码浅析 : 环境准备

一 .前言

本文为准备Flink 1.12.2源码阅读环境.

二 .安装文件准备

• 环境信息如下:

名称	版本
MAC操作系统	11.0.1
JDK	1.8.0_271
Maven	3.6.3
Flink	1.12.2

• 下载flink代码.
  从官方的github下载指定的代码

目前最新的版本为1.12.2版本. (因为官方1.10版本之后有了很大的变动, 建议看1.11版本之后的代码)

我看代码的时候,习惯写备注,为了永久保存和使用方便, 所以代码会提交到github上面.

github地址: https://github.com/BoYiZhang/flink-1.12.2

三 .编译

不墨迹,先给一个可用的,然后再说别的. 官方入口文档

我的scala用的是2.12版本,所以就指定了一下, 默认是2.11.

mvn clean install -DskipTests -Dscala-2.12 -Dcheckstyle.skip=true

这个是在原来的基础上跳过了javadoc

mvn clean install -DskipTests -Dscala-2.12 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true

这个是在上面的基础上跳过了license检查

mvn clean install -DskipTests -Dscala-2.11 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true

然后就是漫长的等待了…

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Flink : 1.12.2:
[INFO]
[INFO] Flink : Tools : Force Shading ...................... SUCCESS [  1.882 s]
[INFO] Flink : ............................................ SUCCESS [ 11.437 s]
[INFO] Flink : Annotations ................................ SUCCESS [  4.826 s]
[INFO] Flink : Test utils : ............................... SUCCESS [  0.473 s]
[INFO] Flink : Test utils : Junit ......................... SUCCESS [  3.072 s]
[INFO] Flink : Metrics : .................................. SUCCESS [  0.177 s]
[INFO] Flink : Metrics : Core ............................. SUCCESS [  2.414 s]
[INFO] Flink : Core ....................................... SUCCESS [ 53.313 s]
[INFO] Flink : Java ....................................... SUCCESS [  9.679 s]
[INFO] Flink : Queryable state : .......................... SUCCESS [  0.156 s]
[INFO] Flink : Queryable state : Client Java .............. SUCCESS [  1.156 s]
[INFO] Flink : FileSystems : .............................. SUCCESS [  0.246 s]
[INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [ 33.051 s]
[INFO] Flink : Runtime .................................... SUCCESS [02:17 min]
[INFO] Flink : Scala ...................................... SUCCESS [01:12 min]
[INFO] Flink : FileSystems : Mapr FS ...................... SUCCESS [  2.796 s]
[INFO] Flink : FileSystems : Hadoop FS shaded ............. SUCCESS [  5.323 s]
[INFO] Flink : FileSystems : S3 FS Base ................... SUCCESS [  9.794 s]
[INFO] Flink : FileSystems : S3 FS Hadoop ................. SUCCESS [  6.568 s]
[INFO] Flink : FileSystems : S3 FS Presto ................. SUCCESS [  9.125 s]
[INFO] Flink : FileSystems : Swift FS Hadoop .............. SUCCESS [ 23.670 s]
[INFO] Flink : FileSystems : OSS FS ....................... SUCCESS [ 10.231 s]
[INFO] Flink : FileSystems : Azure FS Hadoop .............. SUCCESS [01:16 min]
[INFO] Flink : Optimizer .................................. SUCCESS [  7.826 s]
[INFO] Flink : Connectors : ............................... SUCCESS [  0.267 s]
[INFO] Flink : Connectors : File Sink Common .............. SUCCESS [  1.328 s]
[INFO] Flink : Streaming Java ............................. SUCCESS [ 31.717 s]
[INFO] Flink : Clients .................................... SUCCESS [  5.890 s]
[INFO] Flink : Test utils : Utils ......................... SUCCESS [  1.840 s]
[INFO] Flink : Runtime web ................................ SUCCESS [03:55 min]
[INFO] Flink : Examples : ................................. SUCCESS [  0.410 s]
[INFO] Flink : Examples : Batch ........................... SUCCESS [ 23.533 s]
[INFO] Flink : Connectors : Hadoop compatibility .......... SUCCESS [ 18.035 s]
[INFO] Flink : State backends : ........................... SUCCESS [  0.300 s]
[INFO] Flink : State backends : RocksDB ................... SUCCESS [  3.990 s]
[INFO] Flink : Tests ...................................... SUCCESS [01:09 min]
[INFO] Flink : Streaming Scala ............................ SUCCESS [01:11 min]
[INFO] Flink : Connectors : HCatalog ...................... SUCCESS [ 14.497 s]
[INFO] Flink : Test utils : Connectors .................... SUCCESS [  0.533 s]
[INFO] Flink : Connectors : Base .......................... SUCCESS [  1.508 s]
[INFO] Flink : Connectors : Files ......................... SUCCESS [  3.173 s]
[INFO] Flink : Table : .................................... SUCCESS [  0.361 s]
[INFO] Flink : Table : Common ............................. SUCCESS [ 12.452 s]
[INFO] Flink : Table : API Java ........................... SUCCESS [  6.611 s]
[INFO] Flink : Table : API Java bridge .................... SUCCESS [  1.842 s]
[INFO] Flink : Table : API Scala .......................... SUCCESS [ 28.824 s]
[INFO] Flink : Table : API Scala bridge ................... SUCCESS [ 20.010 s]
[INFO] Flink : Table : SQL Parser ......................... SUCCESS [ 35.939 s]
[INFO] Flink : Libraries : ................................ SUCCESS [  0.214 s]
[INFO] Flink : Libraries : CEP ............................ SUCCESS [  7.141 s]
[INFO] Flink : Table : Planner ............................ SUCCESS [03:44 min]
[INFO] Flink : Table : SQL Parser Hive .................... SUCCESS [  6.561 s]
[INFO] Flink : Table : Runtime Blink ...................... SUCCESS [ 16.589 s]
[INFO] Flink : Table : Planner Blink ...................... SUCCESS [05:01 min]
[INFO] Flink : Formats : .................................. SUCCESS [  0.130 s]
[INFO] Flink : Formats : Json ............................. SUCCESS [  1.509 s]
[INFO] Flink : Connectors : Elasticsearch base ............ SUCCESS [ 15.642 s]
[INFO] Flink : Connectors : Elasticsearch 5 ............... SUCCESS [ 12.782 s]
[INFO] Flink : Connectors : Elasticsearch 6 ............... SUCCESS [ 39.903 s]
[INFO] Flink : Connectors : Elasticsearch 7 ............... SUCCESS [  2.080 s]
[INFO] Flink : Connectors : HBase base .................... SUCCESS [  4.462 s]
[INFO] Flink : Connectors : HBase 1.4 ..................... SUCCESS [  4.609 s]
[INFO] Flink : Connectors : HBase 2.2 ..................... SUCCESS [  4.630 s]
[INFO] Flink : Formats : Hadoop bulk ...................... SUCCESS [  2.271 s]
[INFO] Flink : Formats : Orc .............................. SUCCESS [  4.063 s]
[INFO] Flink : Formats : Orc nohive ....................... SUCCESS [  4.200 s]
[INFO] Flink : Formats : Avro ............................. SUCCESS [  6.998 s]
[INFO] Flink : Formats : Parquet .......................... SUCCESS [ 20.453 s]
[INFO] Flink : Formats : Csv .............................. SUCCESS [  2.996 s]
[INFO] Flink : Connectors : Hive .......................... SUCCESS [ 14.302 s]
[INFO] Flink : Connectors : JDBC .......................... SUCCESS [  4.557 s]
[INFO] Flink : Connectors : RabbitMQ ...................... SUCCESS [  1.121 s]
[INFO] Flink : Connectors : Twitter ....................... SUCCESS [  2.545 s]
[INFO] Flink : Connectors : Nifi .......................... SUCCESS [  1.737 s]
[INFO] Flink : Connectors : Cassandra ..................... SUCCESS [  8.202 s]
[INFO] Flink : Metrics : JMX .............................. SUCCESS [  0.636 s]
[INFO] Flink : Connectors : Kafka ......................... SUCCESS [ 54.470 s]
[INFO] Flink : Connectors : Google PubSub ................. SUCCESS [  1.528 s]
[INFO] Flink : Connectors : Kinesis ....................... SUCCESS [ 31.460 s]
[INFO] Flink : Connectors : SQL : Elasticsearch 6 ......... SUCCESS [  9.679 s]
[INFO] Flink : Connectors : SQL : Elasticsearch 7 ......... SUCCESS [ 10.538 s]
[INFO] Flink : Connectors : SQL : HBase 1.4 ............... SUCCESS [ 14.621 s]
[INFO] Flink : Connectors : SQL : HBase 2.2 ............... SUCCESS [ 17.374 s]
[INFO] Flink : Connectors : SQL : Hive 1.2.2 .............. SUCCESS [  5.840 s]
[INFO] Flink : Connectors : SQL : Hive 2.2.0 .............. SUCCESS [  6.194 s]
[INFO] Flink : Connectors : SQL : Hive 2.3.6 .............. SUCCESS [  5.823 s]
[INFO] Flink : Connectors : SQL : Hive 3.1.2 .............. SUCCESS [ 11.041 s]
[INFO] Flink : Connectors : SQL : Kafka ................... SUCCESS [  1.223 s]
[INFO] Flink : Connectors : SQL : Kinesis ................. SUCCESS [ 10.356 s]
[INFO] Flink : Formats : Avro confluent registry .......... SUCCESS [  1.047 s]
[INFO] Flink : Formats : Sequence file .................... SUCCESS [  0.956 s]
[INFO] Flink : Formats : Compress ......................... SUCCESS [  1.072 s]
[INFO] Flink : Formats : SQL Orc .......................... SUCCESS [  0.536 s]
[INFO] Flink : Formats : SQL Parquet ...................... SUCCESS [  0.819 s]
[INFO] Flink : Formats : SQL Avro ......................... SUCCESS [  1.323 s]
[INFO] Flink : Formats : SQL Avro Confluent Registry ...... SUCCESS [ 13.034 s]
[INFO] Flink : Examples : Streaming ....................... SUCCESS [ 22.652 s]
[INFO] Flink : Examples : Table ........................... SUCCESS [ 16.026 s]
[INFO] Flink : Examples : Build Helper : .................. SUCCESS [  0.181 s]
[INFO] Flink : Examples : Build Helper : Streaming Twitter  SUCCESS [  0.788 s]
[INFO] Flink : Examples : Build Helper : Streaming State machine SUCCESS [  0.753 s]
[INFO] Flink : Examples : Build Helper : Streaming Google PubSub SUCCESS [  6.277 s]
[INFO] Flink : Container .................................. SUCCESS [  1.009 s]
[INFO] Flink : Queryable state : Runtime .................. SUCCESS [  1.853 s]
[INFO] Flink : Mesos ...................................... SUCCESS [ 42.373 s]
[INFO] Flink : Kubernetes ................................. SUCCESS [ 18.829 s]
[INFO] Flink : Yarn ....................................... SUCCESS [  3.272 s]
[INFO] Flink : Libraries : Gelly .......................... SUCCESS [  5.959 s]
[INFO] Flink : Libraries : Gelly scala .................... SUCCESS [ 24.488 s]
[INFO] Flink : Libraries : Gelly Examples ................. SUCCESS [ 15.344 s]
[INFO] Flink : External resources : ....................... SUCCESS [  0.108 s]
[INFO] Flink : External resources : GPU ................... SUCCESS [  0.438 s]
[INFO] Flink : Metrics : Dropwizard ....................... SUCCESS [  0.695 s]
[INFO] Flink : Metrics : Graphite ......................... SUCCESS [  0.340 s]
[INFO] Flink : Metrics : InfluxDB ......................... SUCCESS [  1.013 s]
[INFO] Flink : Metrics : Prometheus ....................... SUCCESS [  0.889 s]
[INFO] Flink : Metrics : StatsD ........................... SUCCESS [  0.527 s]
[INFO] Flink : Metrics : Datadog .......................... SUCCESS [  0.469 s]
[INFO] Flink : Metrics : Slf4j ............................ SUCCESS [  0.332 s]
[INFO] Flink : Libraries : CEP Scala ...................... SUCCESS [ 21.172 s]
[INFO] Flink : Table : Uber ............................... SUCCESS [  8.748 s]
[INFO] Flink : Table : Uber Blink ......................... SUCCESS [  6.363 s]
[INFO] Flink : Python ..................................... SUCCESS [ 18.645 s]
[INFO] Flink : Table : SQL Client ......................... SUCCESS [  4.494 s]
[INFO] Flink : Libraries : State processor API ............ SUCCESS [  2.815 s]
[INFO] Flink : ML : ....................................... SUCCESS [  0.214 s]
[INFO] Flink : ML : API ................................... SUCCESS [  0.755 s]
[INFO] Flink : ML : Lib ................................... SUCCESS [  2.303 s]
[INFO] Flink : ML : Uber .................................. SUCCESS [  0.450 s]
[INFO] Flink : Dist ....................................... SUCCESS [ 21.572 s]
[INFO] Flink : Yarn Tests ................................. SUCCESS [  7.607 s]
[INFO] Flink : E2E Tests : ................................ SUCCESS [  1.480 s]
[INFO] Flink : E2E Tests : CLI ............................ SUCCESS [  0.470 s]
[INFO] Flink : E2E Tests : Parent Child classloading program SUCCESS [  0.433 s]
[INFO] Flink : E2E Tests : Parent Child classloading lib-package SUCCESS [  0.405 s]
[INFO] Flink : E2E Tests : Dataset allround ............... SUCCESS [  0.437 s]
[INFO] Flink : E2E Tests : Dataset Fine-grained recovery .. SUCCESS [  0.424 s]
[INFO] Flink : E2E Tests : Datastream allround ............ SUCCESS [  1.420 s]
[INFO] Flink : E2E Tests : Batch SQL ...................... SUCCESS [  0.352 s]
[INFO] Flink : E2E Tests : Stream SQL ..................... SUCCESS [  0.390 s]
[INFO] Flink : E2E Tests : Distributed cache via blob ..... SUCCESS [  0.406 s]
[INFO] Flink : E2E Tests : High parallelism iterations .... SUCCESS [  6.208 s]
[INFO] Flink : E2E Tests : Stream stateful job upgrade .... SUCCESS [  0.811 s]
[INFO] Flink : E2E Tests : Queryable state ................ SUCCESS [  1.647 s]
[INFO] Flink : E2E Tests : Local recovery and allocation .. SUCCESS [  0.447 s]
[INFO] Flink : E2E Tests : Elasticsearch 5 ................ SUCCESS [  5.497 s]
[INFO] Flink : E2E Tests : Elasticsearch 6 ................ SUCCESS [  3.194 s]
[INFO] Flink : Quickstart : ............................... SUCCESS [  2.251 s]
[INFO] Flink : Quickstart : Java .......................... SUCCESS [  0.867 s]
[INFO] Flink : Quickstart : Scala ......................... SUCCESS [  0.366 s]
[INFO] Flink : E2E Tests : Quickstart ..................... SUCCESS [  0.729 s]
[INFO] Flink : E2E Tests : Confluent schema registry ...... SUCCESS [  2.894 s]
[INFO] Flink : E2E Tests : Stream state TTL ............... SUCCESS [  5.564 s]
[INFO] Flink : E2E Tests : SQL client ..................... SUCCESS [  2.014 s]
[INFO] Flink : E2E Tests : File sink ...................... SUCCESS [  1.334 s]
[INFO] Flink : E2E Tests : State evolution ................ SUCCESS [  1.309 s]
[INFO] Flink : E2E Tests : RocksDB state memory control ... SUCCESS [  1.037 s]
[INFO] Flink : E2E Tests : Common ......................... SUCCESS [  2.356 s]
[INFO] Flink : E2E Tests : Metrics availability ........... SUCCESS [  0.451 s]
[INFO] Flink : E2E Tests : Metrics reporter prometheus .... SUCCESS [  0.655 s]
[INFO] Flink : E2E Tests : Heavy deployment ............... SUCCESS [  8.806 s]
[INFO] Flink : E2E Tests : Connectors : Google PubSub ..... SUCCESS [ 17.941 s]
[INFO] Flink : E2E Tests : Streaming Kafka base ........... SUCCESS [  0.549 s]
[INFO] Flink : E2E Tests : Streaming Kafka ................ SUCCESS [  7.020 s]
[INFO] Flink : E2E Tests : Plugins : ...................... SUCCESS [  0.123 s]
[INFO] Flink : E2E Tests : Plugins : Dummy fs ............. SUCCESS [  0.271 s]
[INFO] Flink : E2E Tests : Plugins : Another dummy fs ..... SUCCESS [  0.282 s]
[INFO] Flink : E2E Tests : TPCH ........................... SUCCESS [  0.656 s]
[INFO] Flink : E2E Tests : Streaming Kinesis .............. SUCCESS [ 14.136 s]
[INFO] Flink : E2E Tests : Elasticsearch 7 ................ SUCCESS [  4.222 s]
[INFO] Flink : E2E Tests : Common Kafka ................... SUCCESS [ 10.392 s]
[INFO] Flink : E2E Tests : TPCDS .......................... SUCCESS [  1.618 s]
[INFO] Flink : E2E Tests : Netty shuffle memory control ... SUCCESS [  0.337 s]
[INFO] Flink : E2E Tests : Python ......................... SUCCESS [  9.522 s]
[INFO] Flink : E2E Tests : HBase .......................... SUCCESS [  4.063 s]
[INFO] Flink : State backends : Heap spillable ............ SUCCESS [  1.654 s]
[INFO] Flink : Contrib : .................................. SUCCESS [  0.127 s]
[INFO] Flink : Contrib : Connectors : Wikiedits ........... SUCCESS [  0.999 s]
[INFO] Flink : FileSystems : Tests ........................ SUCCESS [  1.608 s]
[INFO] Flink : Docs ....................................... SUCCESS [  2.829 s]
[INFO] Flink : Walkthrough : .............................. SUCCESS [  0.186 s]
[INFO] Flink : Walkthrough : Common ....................... SUCCESS [  2.009 s]
[INFO] Flink : Walkthrough : Datastream Java .............. SUCCESS [  0.294 s]
[INFO] Flink : Walkthrough : Datastream Scala ............. SUCCESS [  0.227 s]
[INFO] Flink : Tools : CI : Java .......................... SUCCESS [  0.435 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  39:46 min
[INFO] Finished at: 2021-03-10T11:52:37+08:00
[INFO] ------------------------------------------------------------------------

参考文档:
https://ci.apache.org/projects/flink/flink-docs-stable/zh/flinkDev/building.html