一、Flink-1.13.6源码编译运行

一、Flink源码环境搭建

1、概述

本节演示如何在本地编译、运行 Flink 源码。

技术有限，欢迎各位大佬在评论区批评指正。

2、版本说明

名称	版本
flink	1.13.6
jdk	1.8
Maven	3.2.5
操作系统	Mac

3、编译 Flink 源码

1）从 github 下载 Flink 源码

git clone https://github.com/apache/flink

2）切换到目标分支

git checkout remotes/origin/release-1.13

3）编译源码

mvn clean install -DskipTests -Dfast -Dcheckstyle.skip=true

问题：出现某些依赖找不到

解决：单独下载问题 Jar 包，并 install 到本地仓库

问题：依赖下载慢、依赖下载失败

解决：vpn

4）IDEA启动 Standalone 集群

1.配置准备

a）在 flink-dist 模块下 bin 同级目录 创建 log 目录、lib 目录

b）把 resource 目录中的 flink-conf.yaml 文件拷贝到 conf 目录

c）把 target 里的flink-dist.jar包放到 lib 目录，还有四个 log 相关的 jar 包和 zookeeper 相关的 jar 包

2.启动 JobManager

启动类：

flink-runtime 模块下
org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint#main

IDEA 启动参数配置：

VM参数：
-Xmx1073741824
-Xms1073741824
-XX:MaxMetaspaceSize=268435456
-Dlog.file=/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/log/flink-standalonesession.log
-Dlog4j.configuration=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlog4j.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlogback.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/logback-console.xml
-classpath
/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-csv-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-json-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-shaded-zookeeper-3.4.14.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table-blink_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-1.2-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-core-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-slf4j-impl-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/target/flink-dist_2.11-1.13.6.jar:::

运行参数：
--configDir /Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf
--executionMode cluster
-D jobmanager.memory.off-heap.size=134217728b
-D jobmanager.memory.jvm-overhead.min=201326592b
-D jobmanager.memory.jvm-metaspace.size=268435456b
-D jobmanager.memory.heap.size=1073741824b
-D jobmanager.memory.jvm-overhead.max=201326592b

3.启动 TaskManager

启动类：

flink-runtime 模块下
org.apache.flink.runtime.taskexecutor.TaskManagerRunner#main

IDEA 启动参数配置：

VM 参数：
-XX:+UseG1GC
-Xmx536870902
-Xms536870902
-XX:MaxDirectMemorySize=268435458
-XX:MaxMetaspaceSize=268435456
-Dlog.file=/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/log/flink-taskexecutor.log
-Dlog4j.configuration=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlog4j.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/log4j-console.properties
-Dlogback.configurationFile=file:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf/logback-console.xml
-classpath
/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-csv-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-json-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-shaded-zookeeper-3.4.14.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table-blink_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/flink-table_2.11-1.13.6.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-1.2-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-api-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-core-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/src/main/flink-bin/lib/log4j-slf4j-impl-2.17.1.jar:/Users/***/Desktop/flink/flink-dist/target/flink-dist_2.11-1.13.6.jar:::

运行参数：
--configDir /Users/***/Desktop/flink/flink-dist/src/main/flink-bin/conf
-D taskmanager.memory.network.min=134217730b
-D taskmanager.cpu.cores=1.0
-D taskmanager.memory.task.off-heap.size=0b
-D taskmanager.memory.jvm-metaspace.size=268435456b
-D external-resources=none
-D taskmanager.memory.jvm-overhead.min=201326592b
-D taskmanager.memory.framework.off-heap.size=134217728b
-D taskmanager.memory.network.max=134217730b
-D taskmanager.memory.framework.heap.size=134217728b
-D taskmanager.memory.managed.size=536870920b
-D taskmanager.memory.task.heap.size=402653174b
-D taskmanager.numberOfTaskSlots=1
-D taskmanager.memory.jvm-overhead.max=201326592b

问题：运行代码后，打印日志控制台无输出

解决：增加控制台日志配置，在 flink 项目中有 log4j-console.properties 和 logback-console.xml，在 IDEA 启动参数中传入日志配置

-Dlog.file=/Users/*/Desktop/flink-1.13.6/log/flink-standalonesession.log 
-Dlog4j.configuration=file:/Users/*/Desktop/flink-1.13.6/conf/log4j-console.properties 
-Dlog4j.configurationFile=file:/Users/*/Desktop/flink-1.13.6/conf/log4j-console.properties 
-Dlogback.configurationFile=file:/Users/*/Desktop/flink-1.13.6/conf/logback-console.xml

5）打开 Flink WebUI 进行验证

localhost:8081

4、修改源码

1）打包源码

问题：打包源码时，出现某个 Module 找不到

解决：优先打包报错的 Module，并 install 到本地仓库即可

2）修改源码后，重新打包，并运行验证

问题：修改源码后运行无效果，如增加标准输出不显示

解决：每次修改需要重新打包，因为我们运行的是flink-dist模块下的flink-dist_2.11-1.13.6.jar