flink1.17源码编译及Idea下的flink环境

最新推荐文章于 2024-03-29 18:05:05 发布

weixin_42002985

最新推荐文章于 2024-03-29 18:05:05 发布

阅读量1k

点赞数

文章标签： flink

本文链接：https://blog.csdn.net/weixin_42002985/article/details/131506137

版权

官方参考：Building Flink from Source

1、获取源码包：

flink-1.17.1-src.tgz

2、编译打包命令

mvn clean package -DskipTests -Dfast

说明：

-DskipTests：跳过测试

-Dfast：跳过QAplugins、JavaDocs等过程

基于不同的网络环境，package等待中....，最终效果如图

3、基于IDEA本地启动flink环境

本地启动flink环境便于理解FlinkCluster架构，官图如下，至少需启动JobManager和TaskManager

3.1 flink-deamon.sh脚本片段

从bin/flink-deamon.sh脚本中，可以获取对应组件的启动入口

case $DAEMON in
    (taskexecutor)
        CLASS_TO_RUN=org.apache.flink.runtime.taskexecutor.TaskManagerRunner
    ;;

    (zookeeper)
        CLASS_TO_RUN=org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer
    ;;

    (historyserver)
        CLASS_TO_RUN=org.apache.flink.runtime.webmonitor.history.HistoryServer
    ;;

    (standalonesession)
        CLASS_TO_RUN=org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint
    ;;

    (standalonejob)
        CLASS_TO_RUN=org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint
    ;;

    (sql-gateway)
        CLASS_TO_RUN=org.apache.flink.table.gateway.SqlGateway
        SQL_GATEWAY_CLASSPATH="`findSqlGatewayJar`":"`findFlinkPythonJar`"
    ;;

    (*)
        echo "Unknown daemon '${DAEMON}'. $USAGE."
        exit 1
    ;;
esac

3.2 启动JobManager

# ClusterEntrypoint

org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint

# VM options

-Dlog.file=${FLINK_SLOURCE_HOME}/build-target/log/flink-jobmanager-1.local.log
-Dlog4j.configuration=file:${FLINK_SLOURCE_HOME}/build-target/conf/log4j.properties
-Dlog4j.configurationFile=file:${FLINK_SLOURCE_HOME}/build-target/conf/log4j.properties
-Dlogback.configurationFile=file:${FLINK_SLOURCE_HOME}/build-target/conf/logback.xml
-classpath
:${FLINK_SLOURCE_HOME}/build-target/lib/flink-cep-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-connector-files-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-csv-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-dist-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-json-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-scala_2.12-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-table-api-java-uber-1.17.1.jar.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-table-planner-loader-1.17.1.jar.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-table-runtime-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-1.2-api-2.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-api-2.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-core-2.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-slf4j-impl-2.17.1.jar:

-- program arguments

-c ${FLINK_SLOURCE_HOME}/build-target/conf

说明：

${FLINK_SLOURCE_HOME}为实际idea编译构建源代码地址

启动成功后，即可在${FLINK_SLOURCE_HOME}/build-target/log目录下查看到对应的jobmanager日志文件，访问默认8081端口即可查看对应WebUI界面

3.3 启动TaskManager

# 启动入口

org.apache.flink.runtime.taskexecutor.TaskManagerRunner

# VM options

-Dlog.file=${FLINK_SLOURCE_HOME}/build-target/log/flink-taskmanager-1.local.log
-Dlog4j.configuration=file:${FLINK_SLOURCE_HOME}/build-target/conf/log4j.properties
-Dlog4j.configurationFile=file:${FLINK_SLOURCE_HOME}/build-target/conf/log4j.properties
-Dlogback.configurationFile=file:${FLINK_SLOURCE_HOME}/build-target/conf/logback.xml
-classpath
:${FLINK_SLOURCE_HOME}/build-target/lib/flink-cep-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-connector-files-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-csv-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-dist-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-json-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-scala_2.12-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-table-api-java-uber-1.17.1.jar.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-table-planner-loader-1.17.1.jar.jar:${FLINK_SLOURCE_HOME}/build-target/lib/flink-table-runtime-1.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-1.2-api-2.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-api-2.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-core-2.17.1.jar:${FLINK_SLOURCE_HOME}/build-target/lib/log4j-slf4j-impl-2.17.1.jar:

-- program arguments

-c ${FLINK_SLOURCE_HOME}/build-target/conf

-D taskmanager.memory.framework.off-heap.size=134217728b

-D taskmanager.memory.network.max=134217730b

-D taskmanager.memory.network.min=134217730b

-D taskmanager.memory.framework.heap.size=134217728b

-D taskmanager.memory.managed.size=536870920b

-D taskmanager.cpu.cores=4

-D taskmanager.memory.task.heap.size=402653174b

-D taskmanager.memory.task.off-heap.size=0b

-D taskmanager.memory.jvm-metaspace.size=268435456b

-D taskmanager.memory.jvm-overhead.max=201326592b

-D taskmanager.memory.jvm-overhead.min=201326592b