FLINK1.9.1源码编译 支持hadoop2.6.0-cdh1.15.1

官网编译指导:https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#pre-bundled-versions

• 下载源码
  地址:https://flink.apache.org/downloads.html#apache-flink-191
• 编译环境的准备:
  In addition you need Maven 3 and a JDK (Java Development Kit). Flink requires at least Java 8 to build.

[hadoop@hadoop001 flink]$ mvn --version
# maven 版本要求 3.3以上
Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
Maven home: /root/apps/apache-maven-3.6.1
# JDK 1.8.0_191以上
Java version: 1.8.0_221, vendor: Oracle Corporation, runtime: /root/apps/jdk1.8.0_221/jre

• 编译

mvn clean install -DskipTests
cd flink-dist
mvn clean install

• 加上 hadoop cdh 依赖

mvn clean install -DskipTests  -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.15.1 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true

• 下载依赖的 flink-shaded 源码

不同的 Flink 版本使用的 Flink-shaded不同，1.9.1 版本使用 7.0

https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-shaded-7.0/flink-shaded-7.0-src.tgz

• 编译flink-shaded

mvn clean install  -DskipTests -Dhadoop.version=2.6.0-cdh5.15.1

测试 flink on yarn

1. 第一步: 将 /home/hadoop/software/flink-shaded-7.0/flink-shaded-hadoop-2/target/的
   flink-shaded-hadoop-2-2.6.0-cdh5.15.1-7.0.jar 拷贝到编译好的flink的 /home/hadoop/software/flink-1.9.1/build-target/lib路径下

[hadoop@hadoop001 lib]$ ll
total 172700
-rw-r--r-- 1 hadoop hadoop 105319883 Nov 18 15:24 flink-dist_2.11-1.9.1.jar
-rw-r--r-- 1 hadoop hadoop  30101800 Nov 18 15:58 flink-shaded-hadoop-2-2.6.0-cdh5.15.1-7.0.jar
-rw-r--r-- 1 hadoop hadoop  18738069 Nov 18 15:24 flink-table_2.11-1.9.1.jar
-rw-r--r-- 1 hadoop hadoop  22175659 Nov 18 15:24 flink-table-blink_2.11-1.9.1.jar
-rw-r--r-- 1 hadoop hadoop    489884 Oct  8 20:23 log4j-1.2.17.jar
-rw-r--r-- 1 hadoop hadoop      9931 Nov 18 09:46 slf4j-log4j12-1.7.15.jar
[hadoop@hadoop001 lib]$ pwd
/home/hadoop/software/flink-1.9.1/build-target/lib

2. 指定hadoop依赖
   可在系统变量中配置 或在跑yarn 时 执行;

[hadoop@hadoop001 lib]$ export HADOOP_CLASSPATH=`hadoop classpath`
[hadoop@hadoop001 flink]$ bin/flink run -m yarn-cluster -yn 3 -s 4 examples/batch/WordCount.jar

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/software/flink-1.9.1/flink-dist/target/flink-1.9.1-bin/flink-1.9.1/lib/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/apps/hadoop-2.6.0-cdh5.15.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019-11-18 17:13:46,093 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at hadoop001/192.168.0.3:8032
2019-11-18 17:13:46,183 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-11-18 17:13:46,183 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-11-18 17:13:46,186 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-11-18 17:13:46,186 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-11-18 17:13:46,338 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, numberTaskManagers=3, slotsPerTaskManager=1}
2019-11-18 17:13:46,718 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/home/hadoop/software/flink-1.9.1/flink-dist/target/flink-1.9.1-bin/flink-1.9.1/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-11-18 17:13:49,118 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1567577007137_0069
2019-11-18 17:13:49,133 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1567577007137_0069
2019-11-18 17:13:49,133 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-11-18 17:13:49,134 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-11-18 17:13:53,489 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
Starting execution of program
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
(end,2)
(enterprises,1)
(whips,1)
(who,2)
(whose,1)
(will,1)
(wish,1)
(with,3)
(would,2)
(wrong,1)
(you,1)
Program execution finished
Job with JobID f23b0cd07e3144c9090590b7b79a2591 has finished.
Job Runtime: 9711 ms
Accumulator Results: 
- 9aae0b5e4aa683332452de77a9400400 (java.util.ArrayList) [170 elements]