Flink源码编译
step1:环境准备
win10
maven3.2.5(官方推荐)
jdk8
node.js16.6.2
git2.32.0
Flink-src
step2:Flink源码获取
1.github拉取
git clone https://github.com/apache/flink.git
2.官网下载
step3:maven安装
重点:配置阿里云国内镜像
<mirror>
<id>aliyunmaven</id>
<mirrorOf>*</mirrorOf>
<name>阿里云公共仓库</name>
<url>https://maven.aliyun.com/repository/public</url>
</mirror>
step4:编译flink-runtime-web中web-dashboard
由于flink-runtime-web模块在编译时经常卡住报错,所以先把该模块手动编译了,后边整体编译就顺利了。
-
安装node.js
-
注释插件com.github.eirslett
<!-- <plugin>--> <!-- <groupId>com.github.eirslett</groupId>--> <!-- <artifactId>frontend-maven-plugin</artifactId>--> <!-- <version>1.6</version>--> <!-- <executions>--> <!-- <execution>--> <!-- <id>install node and npm</id>--> <!-- <goals>--> <!-- <goal>install-node-and-npm</goal>--> <!-- </goals>--> <!-- <configuration>--> <!-- <nodeVersion>v10.9.0</nodeVersion>--> <!-- </configuration>--> <!-- </execution>--> <!-- <execution>--> <!-- <id>npm install</id>--> <!-- <goals>--> <!-- <goal>npm</goal>--> <!-- </goals>--> <!-- <configuration>--> <!-- <arguments>ci --cache-max=0 --no-save</arguments>--> <!-- <environmentVariables>--> <!-- <HUSKY_SKIP_INSTALL>true</HUSKY_SKIP_INSTALL>--> <!-- </environmentVariables>--> <!-- </configuration>--> <!-- </execution>--> <!-- <execution>--> <!-- <id>npm run build</id>--> <!-- <goals>--> <!-- <goal>npm</goal>--> <!-- </goals>--> <!-- <configuration>--> <!-- <arguments>run build</arguments>--> <!-- </configuration>--> <!-- </execution>--> <!-- </executions>--> <!-- <configuration>--> <!-- <workingDirectory>web-dashboard</workingDirectory>--> <!-- </configuration>--> <!-- </plugin>-->
-
进入flink-runtime-web/web-dashboard文件夹路径下
npm install
npm run build
step5: 整体编译Flink
- 如果你需要使用指定hadoop的版本,可以通过指定“-Dhadoop.version”来设置,编译命令如下:
mvn clean install -DskipTests -Dhadoop.version=2.6.0
# 或者
mvn clean package -DskipTests -Dhadoop.version=2.6.1
- 官方推荐
mvn clean install -DskipTests
-
自用
mvn clean install -T 2C -Dfast -Dmaven.compile.fork=true -DskipTests -Dscala-2.11 -X -Drat.skip=true -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true
-
成功标志
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] force-shading ...................................... SUCCESS [ 1.700 s] [INFO] flink .............................................. SUCCESS [ 1.762 s] [INFO] flink-annotations .................................. SUCCESS [ 3.642 s] [INFO] flink-shaded-curator ............................... SUCCESS [ 3.884 s] [INFO] flink-test-utils-parent ............................ SUCCESS [ 1.998 s] [INFO] flink-test-utils-junit ............................. SUCCESS [ 3.384 s] [INFO] flink-metrics ...................................... SUCCESS [ 2.399 s] [INFO] flink-metrics-core ................................. SUCCESS [ 2.330 s] [INFO] flink-core ......................................... SUCCESS [ 15.151 s] [INFO] flink-java ......................................... SUCCESS [ 6.007 s] [INFO] flink-queryable-state .............................. SUCCESS [ 1.646 s] [INFO] flink-queryable-state-client-java .................. SUCCESS [ 2.140 s] [INFO] flink-filesystems .................................. SUCCESS [ 1.577 s] [INFO] flink-hadoop-fs .................................... SUCCESS [ 3.696 s] [INFO] flink-runtime ...................................... SUCCESS [ 46.766 s] [INFO] flink-scala ........................................ SUCCESS [ 50.829 s] [INFO] flink-mapr-fs ...................................... SUCCESS [ 2.938 s] [INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [ 4.855 s] [INFO] flink-s3-fs-base ................................... SUCCESS [ 3.841 s] [INFO] flink-s3-fs-hadoop ................................. SUCCESS [ 12.073 s] [INFO] flink-s3-fs-presto ................................. SUCCESS [ 16.781 s] [INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 25.736 s] [INFO] flink-oss-fs-hadoop ................................ SUCCESS [ 23.457 s] [INFO] flink-azure-fs-hadoop .............................. SUCCESS [ 16.745 s] [INFO] flink-optimizer .................................... SUCCESS [ 4.232 s] [INFO] flink-clients ...................................... SUCCESS [ 2.655 s] [INFO] flink-streaming-java ............................... SUCCESS [ 12.647 s] [INFO] flink-test-utils ................................... SUCCESS [ 7.587 s] [INFO] flink-runtime-web .................................. SUCCESS [ 18.953 s] [INFO] flink-examples ..................................... SUCCESS [ 0.804 s] [INFO] flink-examples-batch ............................... SUCCESS [ 24.864 s] [INFO] flink-connectors ................................... SUCCESS [ 1.650 s] [INFO] flink-hadoop-compatibility ......................... SUCCESS [ 12.400 s] [INFO] flink-state-backends ............................... SUCCESS [ 1.502 s] [INFO] flink-statebackend-rocksdb ......................... SUCCESS [ 6.749 s] [INFO] flink-tests ........................................ SUCCESS [ 48.688 s] [INFO] flink-streaming-scala .............................. SUCCESS [ 49.604 s] [INFO] flink-table ........................................ SUCCESS [ 1.497 s] [INFO] flink-table-common ................................. SUCCESS [ 5.309 s] [INFO] flink-table-api-java ............................... SUCCESS [ 4.980 s] [INFO] flink-table-api-java-bridge ........................ SUCCESS [ 5.868 s] [INFO] flink-table-api-scala .............................. SUCCESS [ 16.020 s] [INFO] flink-table-api-scala-bridge ....................... SUCCESS [ 15.915 s] [INFO] flink-sql-parser ................................... SUCCESS [ 8.971 s] [INFO] flink-libraries .................................... SUCCESS [ 1.644 s] [INFO] flink-cep .......................................... SUCCESS [ 15.363 s] [INFO] flink-table-planner ................................ SUCCESS [03:10 min] [INFO] flink-table-runtime-blink .......................... SUCCESS [ 7.664 s] [INFO] flink-table-planner-blink .......................... SUCCESS [04:10 min] [INFO] flink-jdbc ......................................... SUCCESS [ 3.730 s] [INFO] flink-hbase ........................................ SUCCESS [ 18.697 s] [INFO] flink-hcatalog ..................................... SUCCESS [ 11.274 s] [INFO] flink-metrics-jmx .................................. SUCCESS [ 5.997 s] [INFO] flink-formats ...................................... SUCCESS [ 1.689 s] [INFO] flink-json ......................................... SUCCESS [ 8.715 s] [INFO] flink-connector-kafka-base ......................... SUCCESS [ 5.394 s] [INFO] flink-connector-kafka-0.9 .......................... SUCCESS [ 6.082 s] [INFO] flink-connector-kafka-0.10 ......................... SUCCESS [ 3.663 s] [INFO] flink-connector-kafka-0.11 ......................... SUCCESS [ 5.391 s] [INFO] flink-connector-elasticsearch-base ................. SUCCESS [ 13.223 s] [INFO] flink-connector-elasticsearch2 ..................... SUCCESS [ 20.612 s] [INFO] flink-connector-elasticsearch5 ..................... SUCCESS [ 21.846 s] [INFO] flink-connector-elasticsearch6 ..................... SUCCESS [ 9.336 s] [INFO] flink-connector-elasticsearch7 ..................... SUCCESS [ 9.337 s] [INFO] flink-orc .......................................... SUCCESS [ 11.241 s] [INFO] flink-csv .......................................... SUCCESS [ 2.528 s] [INFO] flink-connector-hive ............................... SUCCESS [ 28.282 s] [INFO] flink-connector-rabbitmq ........................... SUCCESS [ 5.132 s] [INFO] flink-connector-twitter ............................ SUCCESS [ 7.336 s] [INFO] flink-connector-nifi ............................... SUCCESS [ 2.354 s] [INFO] flink-connector-cassandra .......................... SUCCESS [ 16.468 s] [INFO] flink-avro ......................................... SUCCESS [ 13.278 s] [INFO] flink-connector-filesystem ......................... SUCCESS [ 10.212 s] [INFO] flink-connector-kafka .............................. SUCCESS [ 6.486 s] [INFO] flink-connector-gcp-pubsub ......................... SUCCESS [ 9.487 s] [INFO] flink-connector-kinesis ............................ SUCCESS [ 11.882 s] [INFO] flink-sql-connector-elasticsearch7 ................. SUCCESS [ 11.679 s] [INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [ 10.815 s] [INFO] flink-sql-connector-kafka-0.9 ...................... SUCCESS [ 1.615 s] [INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [ 1.722 s] [INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [ 2.321 s] [INFO] flink-sql-connector-kafka .......................... SUCCESS [ 2.067 s] [INFO] flink-connector-kafka-0.8 .......................... SUCCESS [ 5.224 s] [INFO] flink-avro-confluent-registry ...................... SUCCESS [ 4.200 s] [INFO] flink-parquet ...................................... SUCCESS [ 9.858 s] [INFO] flink-sequence-file ................................ SUCCESS [ 2.365 s] [INFO] flink-compress ..................................... SUCCESS [ 6.189 s] [INFO] flink-examples-streaming ........................... SUCCESS [ 21.874 s] [INFO] flink-examples-table ............................... SUCCESS [ 13.277 s] [INFO] flink-examples-build-helper ........................ SUCCESS [ 0.635 s] [INFO] flink-examples-streaming-twitter ................... SUCCESS [ 1.447 s] [INFO] flink-examples-streaming-state-machine ............. SUCCESS [ 1.183 s] [INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [ 5.948 s] [INFO] flink-container .................................... SUCCESS [ 5.415 s] [INFO] flink-queryable-state-runtime ...................... SUCCESS [ 6.685 s] [INFO] flink-end-to-end-tests ............................. SUCCESS [ 1.505 s] [INFO] flink-cli-test ..................................... SUCCESS [ 3.998 s] [INFO] flink-parent-child-classloading-test-program ....... SUCCESS [ 4.254 s] [INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [ 3.488 s] [INFO] flink-dataset-allround-test ........................ SUCCESS [ 1.711 s] [INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [ 1.954 s] [INFO] flink-datastream-allround-test ..................... SUCCESS [ 6.846 s] [INFO] flink-batch-sql-test ............................... SUCCESS [ 1.494 s] [INFO] flink-stream-sql-test .............................. SUCCESS [ 1.221 s] [INFO] flink-bucketing-sink-test .......................... SUCCESS [ 2.838 s] [INFO] flink-distributed-cache-via-blob ................... SUCCESS [ 3.950 s] [INFO] flink-high-parallelism-iterations-test ............. SUCCESS [ 9.000 s] [INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [ 4.041 s] [INFO] flink-queryable-state-test ......................... SUCCESS [ 6.232 s] [INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [ 2.494 s] [INFO] flink-elasticsearch2-test .......................... SUCCESS [ 2.876 s] [INFO] flink-elasticsearch5-test .......................... SUCCESS [ 3.135 s] [INFO] flink-elasticsearch6-test .......................... SUCCESS [ 7.638 s] [INFO] flink-quickstart ................................... SUCCESS [ 2.395 s] [INFO] flink-quickstart-java .............................. SUCCESS [ 4.188 s] [INFO] flink-quickstart-scala ............................. SUCCESS [ 4.229 s] [INFO] flink-quickstart-test .............................. SUCCESS [ 0.778 s] [INFO] flink-confluent-schema-registry .................... SUCCESS [ 3.102 s] [INFO] flink-stream-state-ttl-test ........................ SUCCESS [ 9.832 s] [INFO] flink-sql-client-test .............................. SUCCESS [ 1.834 s] [INFO] flink-streaming-file-sink-test ..................... SUCCESS [ 4.116 s] [INFO] flink-state-evolution-test ......................... SUCCESS [ 5.834 s] [INFO] flink-rocksdb-state-memory-control-test ............ SUCCESS [ 3.988 s] [INFO] flink-mesos ........................................ SUCCESS [ 40.768 s] [INFO] flink-kubernetes ................................... SUCCESS [ 15.522 s] [INFO] flink-yarn ......................................... SUCCESS [ 10.042 s] [INFO] flink-gelly ........................................ SUCCESS [ 13.021 s] [INFO] flink-gelly-scala .................................. SUCCESS [ 21.494 s] [INFO] flink-gelly-examples ............................... SUCCESS [ 18.450 s] [INFO] flink-metrics-dropwizard ........................... SUCCESS [ 2.049 s] [INFO] flink-metrics-graphite ............................. SUCCESS [ 0.828 s] [INFO] flink-metrics-influxdb ............................. SUCCESS [ 3.054 s] [INFO] flink-metrics-prometheus ........................... SUCCESS [ 2.329 s] [INFO] flink-metrics-statsd ............................... SUCCESS [ 1.905 s] [INFO] flink-metrics-datadog .............................. SUCCESS [ 1.171 s] [INFO] flink-metrics-slf4j ................................ SUCCESS [ 1.896 s] [INFO] flink-cep-scala .................................... SUCCESS [ 18.199 s] [INFO] flink-table-uber ................................... SUCCESS [ 12.974 s] [INFO] flink-table-uber-blink ............................. SUCCESS [ 5.012 s] [INFO] flink-sql-client ................................... SUCCESS [ 5.989 s] [INFO] flink-state-processor-api .......................... SUCCESS [ 2.643 s] [INFO] flink-python ....................................... SUCCESS [ 17.670 s] [INFO] flink-scala-shell .................................. SUCCESS [ 26.648 s] [INFO] flink-dist ......................................... SUCCESS [ 17.640 s] [INFO] flink-end-to-end-tests-common ...................... SUCCESS [ 1.721 s] [INFO] flink-metrics-availability-test .................... SUCCESS [ 1.180 s] [INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [ 1.100 s] [INFO] flink-heavy-deployment-stress-test ................. SUCCESS [ 16.536 s] [INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [ 9.727 s] [INFO] flink-streaming-kafka-test-base .................... SUCCESS [ 4.911 s] [INFO] flink-streaming-kafka-test ......................... SUCCESS [ 9.137 s] [INFO] flink-streaming-kafka011-test ...................... SUCCESS [ 9.769 s] [INFO] flink-streaming-kafka010-test ...................... SUCCESS [ 10.255 s] [INFO] flink-plugins-test ................................. SUCCESS [ 1.490 s] [INFO] dummy-fs ........................................... SUCCESS [ 1.077 s] [INFO] another-dummy-fs ................................... SUCCESS [ 1.101 s] [INFO] flink-tpch-test .................................... SUCCESS [ 4.262 s] [INFO] flink-streaming-kinesis-test ....................... SUCCESS [ 20.838 s] [INFO] flink-elasticsearch7-test .......................... SUCCESS [ 8.038 s] [INFO] flink-end-to-end-tests-common-kafka ................ SUCCESS [ 2.841 s] [INFO] flink-tpcds-test ................................... SUCCESS [ 2.483 s] [INFO] flink-statebackend-heap-spillable .................. SUCCESS [ 2.171 s] [INFO] flink-contrib ...................................... SUCCESS [ 1.579 s] [INFO] flink-connector-wikiedits .......................... SUCCESS [ 5.509 s] [INFO] flink-yarn-tests ................................... SUCCESS [ 7.158 s] [INFO] flink-fs-tests ..................................... SUCCESS [ 7.135 s] [INFO] flink-docs ......................................... SUCCESS [ 4.438 s] [INFO] flink-ml-parent .................................... SUCCESS [ 1.599 s] [INFO] flink-ml-api ....................................... SUCCESS [ 2.633 s] [INFO] flink-ml-lib ....................................... SUCCESS [ 10.818 s] [INFO] flink-walkthroughs ................................. SUCCESS [ 2.391 s] [INFO] flink-walkthrough-common ........................... SUCCESS [ 8.615 s] [INFO] flink-walkthrough-table-java ....................... SUCCESS [ 4.231 s] [INFO] flink-walkthrough-table-scala ...................... SUCCESS [ 4.092 s] [INFO] flink-walkthrough-datastream-java .................. SUCCESS [ 4.293 s] [INFO] flink-walkthrough-datastream-scala ................. SUCCESS [ 4.287 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 09:45 min (Wall Clock) [INFO] Finished at: 2021-08-15T16:12:21+08:00 [INFO] Final Memory: 589M/3489M [INFO] ------------------------------------------------------------------------08:00 [INFO] Final Memory: 589M/3489M [INFO] ------------------------------------------------------------------------ ``
step6:查看编译结果:
编译成功后,编译出完整的flink-binary,在源码目录flink-dist/target/中:
step7:启动本地模式测试
- 运行wordcount测试
- 去webUI查看任务结果