flink 1.9 编译(含cdh版本编译)

一，github获取flink 最新代码

git clone https://github.com/apache/flink

切换到flink 1.9分支

git checkout release-1.9

二，编译flink 代码

flink 集群或单机模式打包，该模式打包是没有yarn-cluster 命令,任务是无法提交任务到yarn上.

mvn clean install -DskipTests -Dfast

flink 包含hadoop 和 yanr-cluster命令打包  

mvn clean install -DskipTests -Drat.skip=true  -Pinclude-hadoop

加入 -Drat.skip=true 可解决如下错误

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project flink-parent: Too many files with unapproved license: 2 See RAT report in: D:\Flink\repositories\flink\target\rat.txt -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :flink-parent

三，cdh平台对应版本打包

如要在其他第三方厂家使用flink 打包，例如要在cdh ,Hortonworks 平台上使用，我们需要指定hadoop 版本进行打包。

例如：

mvn clean install -DskipTests -Pvendor-repos -Pinclude-hadoop -Dhadoop.version=2.6.0-cdh5.16.1

按照上述方法打包可能会找不到 flink-shaded-hadoop-2  如下错误。

[ERROR] Failed to execute goal on project flink-hadoop-fs: Could not resolve dependencies for project org.apache.flink:flink-hadoop-fs:jar:1.9-SNAPSHOT: Failed to collect dependencies at org.apache.flink:flink-shaded-hadoop-2:jar:2.6.0-cdh5.16.1-7.0: Failed to read artifact descriptor for org.apache.flink:flink-shaded-hadoop-2:jar:2.6.0-cdh5.16.1-7.0: Could not transfer artifact org.apache.flink:flink-shaded-hadoop-2:pom:2.6.0-cdh5.16.1-7.0 from/to mapr-releases (https://repository.mapr.com/maven/): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :flink-hadoop-fs

原因是cdh 的 flink-shaded-hadoop-2 的jar 包在mvn 中央仓库是没有编译版本的，我们需要先对flink 的前置依赖 flink-shaded-hadoop-2 进行打包，在进行编译。

获取flink-shaded  git 源码

git clone https://github.com/apache/flink-shaded.git

根据自己缺少的版本切换对应的代码分支，这里我缺少的是7.0版本的flink-shaded-hadoop-2

git checkout  release-7.0

修改flink-shaded项目中的pom.xml 这里修改是为了加入cdh 等中央仓库，否则编译对应版本可能找不到cdh 相关的包。

<profile>
	<id>vendor-repos</id>
	<activation>
		<property>
			<name>vendor-repos</name>
		</property>
	</activation>
	<!-- Add vendor maven repositories -->
	<repositories>
		<!-- Cloudera -->
		<repository>
			<id>cloudera-releases</id>
			<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
			<releases>
				<enabled>true</enabled>
			</releases>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>
		<!-- Hortonworks -->
		<repository>
			<id>HDPReleases</id>
			<name>HDP Releases</name>
			<url>https://repo.hortonworks.com/content/repositories/releases/</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
		<repository>
			<id>HortonworksJettyHadoop</id>
			<name>HDP Jetty</name>
			<url>https://repo.hortonworks.com/content/repositories/jetty-hadoop</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
		<!-- MapR -->
		<repository>
			<id>mapr-releases</id>
			<url>https://repository.mapr.com/maven/</url>
			<snapshots><enabled>false</enabled></snapshots>
			<releases><enabled>true</enabled></releases>
		</repository>
	</repositories>
</profile>

把上述内容加入到<profiles></profiles> 里面去。

编译cdh flink-shaded-hadoop-2

mvn  clean install -DskipTests -Drat.skip=true -Pvendor-repos  -Dhadoop.version=2.6.0-cdh5.16.1

编译完成后在编译flink

mvn clean install -DskipTests -Pvendor-repos -Drat.skip=true -Pinclude-hadoop -Dhadoop.version=2.6.0-cdh5.16.1

三，打包参考文档地址

https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html