一,github获取flink 最新代码
git clone https://github.com/apache/flink
切换到flink 1.9分支
git checkout release-1.9
二,编译flink 代码
flink 集群或单机模式打包,该模式打包是没有yarn-cluster 命令,任务是无法提交任务到yarn上.
mvn clean install -DskipTests -Dfast
flink 包含hadoop 和 yanr-cluster命令打包
mvn clean install -DskipTests -Drat.skip=true -Pinclude-hadoop
加入 -Drat.skip=true 可解决如下错误
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project flink-parent: Too many files with unapproved license: 2 See RAT report in: D:\Flink\repositories\flink\target\rat.txt -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :flink-parent
三,cdh平台对应版本打包
如要在其他第三方厂家使用flink 打包,例如要在cdh ,Hortonworks 平台上使用,我们需要指定hadoop 版本进行打包。
例如:
mvn clean install -DskipTests -Pvendor-repos -Pinclude-hadoop -Dhadoop.version=2.6.0-cdh5.16.1
按照上述方法打包可能会找不到 flink-shaded-hadoop-2 如下错误。
[ERROR] Failed to execute goal on project flink-hadoop-fs: Could not resolve dependencies for project org.apache.flink:flink-hadoop-fs:jar:1.9-SNAPSHOT: Failed to collect dependencies at org.apache.flink:flink-shaded-hadoop-2:jar:2.6.0-cdh5.16.1-7.0: Failed to read artifact descriptor for org.apache.flink:flink-shaded-hadoop-2:jar:2.6.0-cdh5.16.1-7.0: Could not transfer artifact org.apache.flink:flink-shaded-hadoop-2:pom:2.6.0-cdh5.16.1-7.0 from/to mapr-releases (https://repository.mapr.com/maven/): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :flink-hadoop-fs
原因是cdh 的 flink-shaded-hadoop-2 的jar 包在mvn 中央仓库是没有编译版本的,我们需要先对flink 的前置依赖 flink-shaded-hadoop-2 进行打包,在进行编译。
获取flink-shaded git 源码
git clone https://github.com/apache/flink-shaded.git
根据自己缺少的版本切换对应的代码分支,这里我缺少的是7.0版本的flink-shaded-hadoop-2
git checkout release-7.0
修改flink-shaded项目中的pom.xml 这里修改是为了加入cdh 等中央仓库,否则编译对应版本可能找不到cdh 相关的包。
<profile>
<id>vendor-repos</id>
<activation>
<property>
<name>vendor-repos</name>
</property>
</activation>
<!-- Add vendor maven repositories -->
<repositories>
<!-- Cloudera -->
<repository>
<id>cloudera-releases</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<!-- Hortonworks -->
<repository>
<id>HDPReleases</id>
<name>HDP Releases</name>
<url>https://repo.hortonworks.com/content/repositories/releases/</url>
<snapshots><enabled>false</enabled></snapshots>
<releases><enabled>true</enabled></releases>
</repository>
<repository>
<id>HortonworksJettyHadoop</id>
<name>HDP Jetty</name>
<url>https://repo.hortonworks.com/content/repositories/jetty-hadoop</url>
<snapshots><enabled>false</enabled></snapshots>
<releases><enabled>true</enabled></releases>
</repository>
<!-- MapR -->
<repository>
<id>mapr-releases</id>
<url>https://repository.mapr.com/maven/</url>
<snapshots><enabled>false</enabled></snapshots>
<releases><enabled>true</enabled></releases>
</repository>
</repositories>
</profile>
把上述内容加入到<profiles></profiles> 里面去。
编译cdh flink-shaded-hadoop-2
mvn clean install -DskipTests -Drat.skip=true -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.16.1
编译完成后在编译flink
mvn clean install -DskipTests -Pvendor-repos -Drat.skip=true -Pinclude-hadoop -Dhadoop.version=2.6.0-cdh5.16.1
三,打包参考文档地址
https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html