flink1.10.1+hdp3.1.0.0-78编译

基础环境：

[root@hdp2 flink]# which mvn

/root/apache-maven-3.6.3/bin/mvn

 

yum -y install git

 

编译flink1.10.1 集成环境 hdp3.1.0.0-78

 

 

https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/comp_versions.html

 

[root@hdp2 flink-shaded]# mvn clean install -DskipTests -Dhadoop.version=3.1.1.3.1.0.0-78 -Pvendor-repos

 

先报错 hadoop-fs模块部分

 

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <args> -rf :flink-hadoop-fs

 

 

[root@hdp2 flink-shaded]# mvn clean install

 

 

编译flink-shaded 源码   说明：hadoop版本：Apache Hadoop 3.1.1  HDP版本：3.1.0.0-78

 

[root@hdp2 flink-shaded]# mvn clean install -DskipTests -Dhadoop.version=3.1.1.3.1.0.0-78 -Pvendor-repos

[INFO]

[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ flink-shaded-hadoop-2-uber ---

[INFO] Installing /root/flink-shaded/flink-shaded-hadoop-2-uber/target/flink-shaded-hadoop-2-uber-3.1.1.3.1.0.0-78-9.0.jar to /root/.m2/repository/org/apache/flink/flink-shaded-hadoop-2-uber/3.1.1.3.1.0.0-78-9.0/flink-shaded-hadoop-2-uber-3.1.1.3.1.0.0-78-9.0.jar

[INFO] Installing /root/flink-shaded/flink-shaded-hadoop-2-uber/target/dependency-reduced-pom.xml to /root/.m2/repository/org/apache/flink/flink-shaded-hadoop-2-uber/3.1.1.3.1.0.0-78-9.0/flink-shaded-hadoop-2-uber-3.1.1.3.1.0.0-78-9.0.pom

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] flink-shaded 9.0 ................................... SUCCESS [  2.767 s]

[INFO] flink-shaded-force-shading 9.0 ..................... SUCCESS [  0.735 s]

[INFO] flink-shaded-asm-7 7.1-9.0 ......................... SUCCESS [  0.773 s]

[INFO] flink-shaded-guava-18 18.0-9.0 ..................... SUCCESS [  1.398 s]

[INFO] flink-shaded-netty-4 4.1.39.Final-9.0 .............. SUCCESS [  4.374 s]

[INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-9.0 SUCCESS [  0.770 s]

[INFO] flink-shaded-jackson-parent 2.10.1-9.0 ............. SUCCESS [  0.023 s]

[INFO] flink-shaded-jackson-2 2.10.1-9.0 .................. SUCCESS [  1.855 s]

[INFO] flink-shaded-jackson-module-jsonSchema-2 2.10.1-9.0  SUCCESS [  1.060 s]

[INFO] flink-shaded-hadoop-2 3.1.1.3.1.0.0-78-9.0 ......... SUCCESS [ 28.549 s]

[INFO] flink-shaded-hadoop-2-uber 3.1.1.3.1.0.0-78-9.0 .... SUCCESS [ 39.257 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  01:21 min

[INFO] Finished at: 2020-07-06T17:32:52+08:00

[INFO] ------------------------------------------------------------------------

[root@hdp2 flink-shaded]#

 

 

说明：hadoop版本：Apache Hadoop 3.1.1  HDP版本：3.1.0.0-78

 

说明： /root/flink/pom.xml 中查看依赖的flink-shaded版本： <flink.shaded.version>9.0</flink.shaded.version>

 

[root@hdp2 flink]# mvn clean install -DskipTests -Dhadoop.version=3.1.1.3.1.0.0-78 -Pvendor-repos

 

 

[INFO] Installing /root/flink/flink-walkthroughs/flink-walkthrough-datastream-scala/target/flink-walkthrough-datastream-scala-1.10.1-tests.jar to /root/.m2/repository/org/apache/flink/flink-walkthrough-datastream-scala/1.10.1/flink-walkthrough-datastream-scala-1.10.1-tests.jar

[INFO]

[INFO] --- maven-archetype-plugin:2.2:update-local-catalog (default-update-local-catalog) @ flink-walkthrough-datastream-scala ---

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary for flink 1.10.1:

[INFO]

[INFO] force-shading ...................................... SUCCESS [  1.930 s]

[INFO] flink .............................................. SUCCESS [  8.228 s]

[INFO] flink-annotations .................................. SUCCESS [  1.836 s]

[INFO] flink-shaded-curator ............................... SUCCESS [  1.508 s]

[INFO] flink-test-utils-parent ............................ SUCCESS [  0.158 s]

[INFO] flink-test-utils-junit ............................. SUCCESS [  1.112 s]

[INFO] flink-metrics ...................................... SUCCESS [  0.171 s]

[INFO] flink-metrics-core ................................. SUCCESS [  1.976 s]

[INFO] flink-core ......................................... SUCCESS [ 55.902 s]

[INFO] flink-java ......................................... SUCCESS [  8.150 s]

[INFO] flink-queryable-state .............................. SUCCESS [  0.130 s]

[INFO] flink-queryable-state-client-java .................. SUCCESS [  1.063 s]

[INFO] flink-filesystems .................................. SUCCESS [  0.132 s]

[INFO] flink-hadoop-fs .................................... SUCCESS [ 12.684 s]

[INFO] flink-runtime ...................................... SUCCESS [03:11 min]

[INFO] flink-scala ........................................ SUCCESS [01:01 min]

[INFO] flink-mapr-fs ...................................... SUCCESS [  0.903 s]

[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [  6.263 s]

[INFO] flink-s3-fs-base ................................... SUCCESS [  1.246 s]

[INFO] flink-s3-fs-hadoop ................................. SUCCESS [  6.953 s]

[INFO] flink-s3-fs-presto ................................. SUCCESS [ 10.393 s]

[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 26.159 s]

[INFO] flink-oss-fs-hadoop ................................ SUCCESS [  8.563 s]

[INFO] flink-azure-fs-hadoop .............................. SUCCESS [ 34.443 s]

[INFO] flink-optimizer .................................... SUCCESS [ 31.418 s]

[INFO] flink-clients ...................................... SUCCESS [  2.506 s]

[INFO] flink-streaming-java ............................... SUCCESS [ 19.954 s]

[INFO] flink-test-utils ................................... SUCCESS [  1.532 s]

[INFO] flink-runtime-web .................................. SUCCESS [02:41 min]

[INFO] flink-examples ..................................... SUCCESS [  0.216 s]

[INFO] flink-examples-batch ............................... SUCCESS [ 18.960 s]

[INFO] flink-connectors ................................... SUCCESS [  0.139 s]

[INFO] flink-hadoop-compatibility ......................... SUCCESS [  7.260 s]

[INFO] flink-state-backends ............................... SUCCESS [  0.135 s]

[INFO] flink-statebackend-rocksdb ......................... SUCCESS [  3.164 s]

[INFO] flink-tests ........................................ SUCCESS [ 56.056 s]

[INFO] flink-streaming-scala .............................. SUCCESS [ 49.707 s]

[INFO] flink-table ........................................ SUCCESS [  0.120 s]

[INFO] flink-table-common ................................. SUCCESS [  5.820 s]

[INFO] flink-table-api-java ............................... SUCCESS [  3.465 s]

[INFO] flink-table-api-java-bridge ........................ SUCCESS [  1.346 s]

[INFO] flink-table-api-scala .............................. SUCCESS [ 14.149 s]

[INFO] flink-table-api-scala-bridge ....................... SUCCESS [ 14.223 s]

[INFO] flink-sql-parser ................................... SUCCESS [ 11.298 s]

[INFO] flink-libraries .................................... SUCCESS [  0.120 s]

[INFO] flink-cep .......................................... SUCCESS [  4.279 s]

[INFO] flink-table-planner ................................ SUCCESS [03:08 min]

[INFO] flink-table-runtime-blink .......................... SUCCESS [  9.121 s]

[INFO] flink-table-planner-blink .......................... SUCCESS [04:43 min]

[INFO] flink-jdbc ......................................... SUCCESS [  1.048 s]

[INFO] flink-hbase ........................................ SUCCESS [ 52.687 s]

[INFO] flink-hcatalog ..................................... SUCCESS [  7.981 s]

[INFO] flink-metrics-jmx .................................. SUCCESS [  0.513 s]

[INFO] flink-formats ...................................... SUCCESS [  0.134 s]

[INFO] flink-json ......................................... SUCCESS [  0.864 s]

[INFO] flink-connector-kafka-base ......................... SUCCESS [  3.314 s]

[INFO] flink-connector-kafka-0.9 .......................... SUCCESS [  1.812 s]

[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [  1.178 s]

[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [  1.518 s]

[INFO] flink-connector-elasticsearch-base ................. SUCCESS [  1.571 s]

[INFO] flink-connector-elasticsearch2 ..................... SUCCESS [ 16.409 s]

[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [ 18.343 s]

[INFO] flink-connector-elasticsearch6 ..................... SUCCESS [  1.370 s]

[INFO] flink-connector-elasticsearch7 ..................... SUCCESS [  1.301 s]

[INFO] flink-orc .......................................... SUCCESS [  1.327 s]

[INFO] flink-csv .......................................... SUCCESS [  0.538 s]

[INFO] flink-connector-hive ............................... SUCCESS [  5.555 s]

[INFO] flink-connector-rabbitmq ........................... SUCCESS [  0.584 s]

[INFO] flink-connector-twitter ............................ SUCCESS [  2.612 s]

[INFO] flink-connector-nifi ............................... SUCCESS [  0.675 s]

[INFO] flink-connector-cassandra .......................... SUCCESS [  4.581 s]

[INFO] flink-avro ......................................... SUCCESS [  3.147 s]

[INFO] flink-connector-filesystem ......................... SUCCESS [  1.408 s]

[INFO] flink-connector-kafka .............................. SUCCESS [  1.321 s]

[INFO] flink-connector-gcp-pubsub ......................... SUCCESS [ 16.839 s]

[INFO] flink-connector-kinesis ............................ SUCCESS [ 17.805 s]

[INFO] flink-sql-connector-elasticsearch7 ................. SUCCESS [ 10.935 s]

[INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [  8.862 s]

[INFO] flink-sql-connector-kafka-0.9 ...................... SUCCESS [  0.443 s]

[INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [  0.643 s]

[INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [  0.765 s]

[INFO] flink-sql-connector-kafka .......................... SUCCESS [  1.263 s]

[INFO] flink-connector-kafka-0.8 .......................... SUCCESS [  1.200 s]

[INFO] flink-avro-confluent-registry ...................... SUCCESS [  1.792 s]

[INFO] flink-parquet ...................................... SUCCESS [  1.548 s]

[INFO] flink-sequence-file ................................ SUCCESS [  0.413 s]

[INFO] flink-compress ..................................... SUCCESS [  0.458 s]

[INFO] flink-examples-streaming ........................... SUCCESS [ 21.754 s]

[INFO] flink-examples-table ............................... SUCCESS [ 11.955 s]

[INFO] flink-examples-build-helper ........................ SUCCESS [  0.204 s]

[INFO] flink-examples-streaming-twitter ................... SUCCESS [  0.983 s]

[INFO] flink-examples-streaming-state-machine ............. SUCCESS [  0.609 s]

[INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [  6.735 s]

[INFO] flink-container .................................... SUCCESS [  0.673 s]

[INFO] flink-queryable-state-runtime ...................... SUCCESS [  1.116 s]

[INFO] flink-end-to-end-tests ............................. SUCCESS [  0.107 s]

[INFO] flink-cli-test ..................................... SUCCESS [  0.376 s]

[INFO] flink-parent-child-classloading-test-program ....... SUCCESS [  0.300 s]

[INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [  0.206 s]

[INFO] flink-dataset-allround-test ........................ SUCCESS [  0.278 s]

[INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [  0.274 s]

[INFO] flink-datastream-allround-test ..................... SUCCESS [  2.075 s]

[INFO] flink-batch-sql-test ............................... SUCCESS [  0.277 s]

[INFO] flink-stream-sql-test .............................. SUCCESS [  0.323 s]

[INFO] flink-bucketing-sink-test .......................... SUCCESS [  0.676 s]

[INFO] flink-distributed-cache-via-blob ................... SUCCESS [  0.277 s]

[INFO] flink-high-parallelism-iterations-test ............. SUCCESS [  9.793 s]

[INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [  1.249 s]

[INFO] flink-queryable-state-test ......................... SUCCESS [  2.244 s]

[INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [  0.533 s]

[INFO] flink-elasticsearch2-test .......................... SUCCESS [  5.323 s]

[INFO] flink-elasticsearch5-test .......................... SUCCESS [  7.515 s]

[INFO] flink-elasticsearch6-test .......................... SUCCESS [  3.737 s]

[INFO] flink-quickstart ................................... SUCCESS [  1.730 s]

[INFO] flink-quickstart-java .............................. SUCCESS [  0.800 s]

[INFO] flink-quickstart-scala ............................. SUCCESS [  0.275 s]

[INFO] flink-quickstart-test .............................. SUCCESS [  0.721 s]

[INFO] flink-confluent-schema-registry .................... SUCCESS [  2.325 s]

[INFO] flink-stream-state-ttl-test ........................ SUCCESS [  5.560 s]

[INFO] flink-sql-client-test .............................. SUCCESS [  0.799 s]

[INFO] flink-streaming-file-sink-test ..................... SUCCESS [  0.314 s]

[INFO] flink-state-evolution-test ......................... SUCCESS [  1.183 s]

[INFO] flink-rocksdb-state-memory-control-test ............ SUCCESS [  1.110 s]

[INFO] flink-mesos ........................................ SUCCESS [ 32.418 s]

[INFO] flink-kubernetes ................................... SUCCESS [  6.587 s]

[INFO] flink-yarn ......................................... SUCCESS [  2.023 s]

[INFO] flink-gelly ........................................ SUCCESS [  4.440 s]

[INFO] flink-gelly-scala .................................. SUCCESS [ 26.528 s]

[INFO] flink-gelly-examples ............................... SUCCESS [ 14.610 s]

[INFO] flink-metrics-dropwizard ........................... SUCCESS [  0.488 s]

[INFO] flink-metrics-graphite ............................. SUCCESS [  0.270 s]

[INFO] flink-metrics-influxdb ............................. SUCCESS [  1.193 s]

[INFO] flink-metrics-prometheus ........................... SUCCESS [  0.635 s]

[INFO] flink-metrics-statsd ............................... SUCCESS [  0.468 s]

[INFO] flink-metrics-datadog .............................. SUCCESS [  0.565 s]

[INFO] flink-metrics-slf4j ................................ SUCCESS [  0.387 s]

[INFO] flink-cep-scala .................................... SUCCESS [ 17.226 s]

[INFO] flink-table-uber ................................... SUCCESS [  3.368 s]

[INFO] flink-table-uber-blink ............................. SUCCESS [  3.485 s]

[INFO] flink-sql-client ................................... SUCCESS [  2.388 s]

[INFO] flink-state-processor-api .......................... SUCCESS [  1.287 s]

[INFO] flink-python ....................................... SUCCESS [ 14.974 s]

[INFO] flink-scala-shell .................................. SUCCESS [ 17.179 s]

[INFO] flink-dist ......................................... SUCCESS [ 23.667 s]

[INFO] flink-end-to-end-tests-common ...................... SUCCESS [  1.126 s]

[INFO] flink-metrics-availability-test .................... SUCCESS [  0.306 s]

[INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [  0.297 s]

[INFO] flink-heavy-deployment-stress-test ................. SUCCESS [  9.647 s]

[INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [  1.114 s]

[INFO] flink-streaming-kafka-test-base .................... SUCCESS [  0.434 s]

[INFO] flink-streaming-kafka-test ......................... SUCCESS [  9.287 s]

[INFO] flink-streaming-kafka011-test ...................... SUCCESS [  8.080 s]

[INFO] flink-streaming-kafka010-test ...................... SUCCESS [  9.066 s]

[INFO] flink-plugins-test ................................. SUCCESS [  0.129 s]

[INFO] dummy-fs ........................................... SUCCESS [  0.199 s]

[INFO] another-dummy-fs ................................... SUCCESS [  0.195 s]

[INFO] flink-tpch-test .................................... SUCCESS [  1.112 s]

[INFO] flink-streaming-kinesis-test ....................... SUCCESS [ 14.524 s]

[INFO] flink-elasticsearch7-test .......................... SUCCESS [  4.693 s]

[INFO] flink-end-to-end-tests-common-kafka ................ SUCCESS [  1.736 s]

[INFO] flink-tpcds-test ................................... SUCCESS [  1.059 s]

[INFO] flink-statebackend-heap-spillable .................. SUCCESS [  1.220 s]

[INFO] flink-contrib ...................................... SUCCESS [  0.118 s]

[INFO] flink-connector-wikiedits .......................... SUCCESS [  0.500 s]

[INFO] flink-yarn-tests ................................... SUCCESS [  6.109 s]

[INFO] flink-fs-tests ..................................... SUCCESS [  0.934 s]

[INFO] flink-docs ......................................... SUCCESS [  1.406 s]

[INFO] flink-ml-parent .................................... SUCCESS [  0.099 s]

[INFO] flink-ml-api ....................................... SUCCESS [  0.630 s]

[INFO] flink-ml-lib ....................................... SUCCESS [  1.301 s]

[INFO] flink-walkthroughs ................................. SUCCESS [  0.170 s]

[INFO] flink-walkthrough-common ........................... SUCCESS [  1.405 s]

[INFO] flink-walkthrough-table-java ....................... SUCCESS [  0.242 s]

[INFO] flink-walkthrough-table-scala ...................... SUCCESS [  0.315 s]

[INFO] flink-walkthrough-datastream-java .................. SUCCESS [  0.209 s]

[INFO] flink-walkthrough-datastream-scala ................. SUCCESS [  0.203 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  31:14 min

[INFO] Finished at: 2020-07-07T10:05:30+08:00

[INFO] ------------------------------------------------------------------------

[root@hdp2 flink]#

 

编译后的源码：

[root@hdp2 flink-1.10.1-bin]# pwd

/root/flink/flink-dist/target/flink-1.10.1-bin

[root@hdp2 flink-1.10.1-bin]# ls

flink-1.10.1