基于CDH-6.1.0编译flink-1.11-SNAPSHOT

最新推荐文章于 2023-05-25 21:13:11 发布
最新推荐文章于 2023-05-25 21:13:11 发布
阅读量4.3k 收藏 7
点赞数 2
分类专栏： Flink 环境搭建 文章标签： flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u014310499/article/details/104006346
版权

编译准备环境：

  • jdk-1.8
  • maven-3.2.5

下载flink-1.11安装包

git clone --branch master https://github.com/apache/flink.git

执行编译命令：

mvn -T2C clean install -DskipTests -Dfast -Pinclude-hadoop -Pvendor-repos -Dhadoop.version=3.0.0-cdh6.1.0 -Dscala-2.11

执行报错：

[ERROR] Failed to execute goal on project flink-hadoop-fs: Could not resolve dependencies for project org.apache.flink:flink-hadoop-fs:jar:1.11-SNAPSHOT: The following artifacts could not be resolved: org.apach
e.flink:flink-shaded-hadoop-2:jar:3.0.0-cdh6.1.0-9.0, org.apache.hadoop:hadoop-hdfs:jar:tests:3.0.0-cdh6.1.0, org.apache.hadoop:hadoop-common:jar:tests:3.0.0-cdh6.1.0: Could not find artifact org.apache.flink:f
link-shaded-hadoop-2:jar:3.0.0-cdh6.1.0-9.0 in alimaven (http://maven.aliyun.com/nexus/content/groups/public/) -> [Help 1]

这里缺少flink-shaded-hadoop 9.0，下载并编译：

CDH-6.1.0的Hadoop版本：Hadoop 3.0.0-cdh6.1.0

开始编译：

mvn clean install -DskipTests -Dhadoop.version=3.0.0-cdh6.1.0

在 flink-shaded-9.0/pom.xml文件中添加 cloudera 的maven库：

<!--添加CDH的仓库-->
<repositories>
	<repository>
		<id>cloudera</id>
		<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
	</repository>
</repositories>

编译成功：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] flink-shaded ....................................... SUCCESS [  2.267 s]
[INFO] flink-shaded-force-shading ......................... SUCCESS [  0.647 s]
[INFO] flink-shaded-asm-7 ................................. SUCCESS [  0.928 s]
[INFO] flink-shaded-guava-18 .............................. SUCCESS [  2.467 s]
[INFO] flink-shaded-netty-4 ............................... SUCCESS [ 20.499 s]
[INFO] flink-shaded-netty-tcnative-dynamic ................ SUCCESS [  1.570 s]
[INFO] flink-shaded-jackson-parent ........................ SUCCESS [  0.073 s]
[INFO] flink-shaded-jackson-2 ............................. SUCCESS [  1.314 s]
[INFO] flink-shaded-jackson-module-jsonSchema-2 ........... SUCCESS [  0.917 s]
[INFO] flink-shaded-hadoop-2 .............................. SUCCESS [ 15.471 s]
[INFO] flink-shaded-hadoop-2-uber ......................... SUCCESS [ 17.707 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:04 min
[INFO] Finished at: 2020-01-15T15:49:45+08:00
[INFO] Final Memory: 52M/1423M
[INFO] ------------------------------------------------------------------------

再次编译flink，执行报错：

[ERROR] Failed to execute goal on project flink-hadoop-fs: Could not resolve dependencies for project org.apache.flink:flink-hadoop-fs:jar:1.11-SNAPSHOT: The following artifacts could not be resolved: org.apach
e.hadoop:hadoop-hdfs:jar:tests:3.0.0-cdh6.1.0, org.apache.hadoop:hadoop-common:jar:tests:3.0.0-cdh6.1.0: Failure to find org.apache.hadoop:hadoop-hdfs:jar:tests:3.0.0-cdh6.1.0 in http://maven.aliyun.com/nexus/c
ontent/groups/public/ was cached in the local repository, resolution will not be reattempted until the update interval of alimaven has elapsed or updates are forced -> [Help 1]

在flink-1.11的pom.xml文件中也加入cloudera的库：

<!--添加CDH的仓库-->
<repositories>
	<repository>
		<id>cloudera</id>
		<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
	</repository>
</repositories>

再次编译，报错：

[ERROR] /G:/Projects/IdeaProjects/flink-1.11/flink-master/flink-yarn/src/test/java/org/apache/flink/yarn/AbstractYarnClusterTest.java:[85,41] 对于newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org
.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,<nulltype>,org.apache.hadoop.yarn.api.records.YarnApplicationState,<nulltype>,<nullty
pe>,long,long,org.apache.hadoop.yarn.api.records.FinalApplicationStatus,<nulltype>,<nulltype>,float,<nulltype>,<nulltype>), 找不到合适的方法
[ERROR] 方法 org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.
String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,long,org.apache.hadoop.y
arn.api.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token)不适用
[ERROR] (实际参数列表和形式参数列表长度不同)
[ERROR] 方法 org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.
String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,org.apache.hadoop.yarn.a
pi.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token,java.util.Set<java.lang.Strin
g>,boolean,org.apache.hadoop.yarn.api.records.Priority,java.lang.String,java.lang.String)不适用
[ERROR] (实际参数列表和形式参数列表长度不同)
[ERROR] 方法 org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId,org.apache.hadoop.yarn.api.records.ApplicationAttemptId,java.lang.String,java.lang.
String,java.lang.String,java.lang.String,int,org.apache.hadoop.yarn.api.records.Token,org.apache.hadoop.yarn.api.records.YarnApplicationState,java.lang.String,java.lang.String,long,long,long,org.apache.hadoop.y
arn.api.records.FinalApplicationStatus,org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport,java.lang.String,float,java.lang.String,org.apache.hadoop.yarn.api.records.Token,java.util.Set<java.lang.
String>,boolean,org.apache.hadoop.yarn.api.records.Priority,java.lang.String,java.lang.String)不适用
[ERROR] (实际参数列表和形式参数列表长度不同)

可以看到flink-yarn的测试项目发生了异常，在这个版本中org.apache.hadoop.yarn.api.records.ApplicationReport.newInstance方法是不可用的，因此我们需要在flink-yarn模块下的pom文件的build中添加如下插件，跳过本模块的测试代码的编译。

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.0</version>
        <configuration>
            <source>${java.version}</source>
            <target>${java.version}</target>
            <!-- 略过测试代码的编译 -->
            <skip>true</skip>
            <!-- The semantics of this option are reversed, see MCOMPILER-209. -->
            <useIncrementalCompilation>false</useIncrementalCompilation>
            <compilerArgs>
                 <!-- Prevents recompilation due to missing package-info.class, see MCOMPILER-205 -->
                <arg>-Xpkginfo:always</arg>
            </compilerArgs>
        </configuration>
    </plugin>

这里在YARN集群运行时不会受到影响。

参考：https://blog.csdn.net/github_39577257/article/details/97648316

再次编译，报错：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile (default-testCompile) on project flink-yarn-tests: Compilation failure: Compilation failure:
[ERROR] /G:/Projects/IdeaProjects/flink-1.11/flink-master/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnPrioritySchedulingITCase.java:[28,36] 找不到符号
[ERROR] 符号:   类 YarnTestUtils
[ERROR] 位置: 程序包 org.apache.flink.yarn
[ERROR] /G:/Projects/IdeaProjects/flink-1.11/flink-master/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnPrioritySchedulingITCase.java:[28,1] 仅从类和接口静态导入
[ERROR] /G:/Projects/IdeaProjects/flink-1.11/flink-master/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java:[308,69] 找不到符号
[ERROR] 符号:   变量 YarnTestUtils
[ERROR] 位置: 类 org.apache.flink.yarn.YarnTestBase
[ERROR] /G:/Projects/IdeaProjects/flink-1.11/flink-master/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnPrioritySchedulingITCase.java:[42,25] 找不到符号
[ERROR] 符号:   方法 isHadoopVersionGreaterThanOrEquals(int,int)
[ERROR] 位置: 类 org.apache.flink.yarn.YarnPrioritySchedulingITCase
[ERROR] /G:/Projects/IdeaProjects/flink-1.11/flink-master/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnConfigurationITCase.java:[94,73] 找不到符号
[ERROR] 符号:   变量 YarnTestUtils
[ERROR] 位置: 类 org.apache.flink.yarn.YarnConfigurationITCase

同样在flink-yarn-test模块下的pom.xml中添加：

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.0</version>
        <configuration>
            <source>${java.version}</source>
            <target>${java.version}</target>
            <!-- 略过测试代码的编译 -->
            <skip>true</skip>
            <!-- The semantics of this option are reversed, see MCOMPILER-209. -->
            <useIncrementalCompilation>false</useIncrementalCompilation>
            <compilerArgs>
                 <!-- Prevents recompilation due to missing package-info.class, see MCOMPILER-205 -->
                <arg>-Xpkginfo:always</arg>
            </compilerArgs>
        </configuration>
    </plugin>

编译成功！

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] force-shading ...................................... SUCCESS [  3.364 s]
[INFO] flink .............................................. SUCCESS [  2.476 s]
[INFO] flink-annotations .................................. SUCCESS [  5.092 s]
[INFO] flink-shaded-curator ............................... SUCCESS [  4.777 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [  0.775 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [  6.614 s]
[INFO] flink-metrics ...................................... SUCCESS [  0.959 s]
[INFO] flink-metrics-core ................................. SUCCESS [  2.883 s]
[INFO] flink-core ......................................... SUCCESS [ 39.715 s]
[INFO] flink-java ......................................... SUCCESS [ 15.704 s]
[INFO] flink-queryable-state .............................. SUCCESS [  0.622 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [  3.589 s]
[INFO] flink-filesystems .................................. SUCCESS [  0.577 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [  6.644 s]
[INFO] flink-runtime ...................................... SUCCESS [01:43 min]
[INFO] flink-scala ........................................ SUCCESS [01:25 min]
[INFO] flink-mapr-fs ...................................... SUCCESS [  4.183 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [  7.549 s]
[INFO] flink-s3-fs-base ................................... SUCCESS [  5.433 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [ 11.240 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [ 17.500 s]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 33.569 s]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [ 23.678 s]
[INFO] flink-azure-fs-hadoop .............................. SUCCESS [ 21.042 s]
[INFO] flink-optimizer .................................... SUCCESS [ 10.199 s]
[INFO] flink-clients ...................................... SUCCESS [  5.264 s]
[INFO] flink-streaming-java ............................... SUCCESS [ 43.541 s]
[INFO] flink-test-utils ................................... SUCCESS [  5.967 s]
[INFO] flink-runtime-web .................................. SUCCESS [04:13 min]
[INFO] flink-examples ..................................... SUCCESS [  0.804 s]
[INFO] flink-examples-batch ............................... SUCCESS [01:23 min]
[INFO] flink-connectors ................................... SUCCESS [  0.577 s]
[INFO] flink-hadoop-compatibility ......................... SUCCESS [ 27.781 s]
[INFO] flink-state-backends ............................... SUCCESS [  0.774 s]
[INFO] flink-statebackend-rocksdb ......................... SUCCESS [ 11.679 s]
[INFO] flink-tests ........................................ SUCCESS [01:17 min]
[INFO] flink-streaming-scala .............................. SUCCESS [01:18 min]
[INFO] flink-table ........................................ SUCCESS [  0.782 s]
[INFO] flink-table-common ................................. SUCCESS [ 15.568 s]
[INFO] flink-table-api-java ............................... SUCCESS [ 14.810 s]
[INFO] flink-table-api-java-bridge ........................ SUCCESS [  4.774 s]
[INFO] flink-table-api-scala .............................. SUCCESS [ 15.980 s]
[INFO] flink-table-api-scala-bridge ....................... SUCCESS [ 16.605 s]
[INFO] flink-sql-parser ................................... SUCCESS [ 15.369 s]
[INFO] flink-libraries .................................... SUCCESS [  0.525 s]
[INFO] flink-cep .......................................... SUCCESS [ 58.830 s]
[INFO] flink-table-planner ................................ SUCCESS [04:41 min]
[INFO] flink-jdbc ......................................... SUCCESS [ 12.326 s]
[INFO] flink-table-runtime-blink .......................... SUCCESS [ 44.744 s]
[INFO] flink-table-planner-blink .......................... SUCCESS [07:07 min]
[INFO] flink-hbase ........................................ SUCCESS [ 15.265 s]
[INFO] flink-hcatalog ..................................... SUCCESS [ 37.772 s]
[INFO] flink-metrics-jmx .................................. SUCCESS [  4.656 s]
[INFO] flink-formats ...................................... SUCCESS [  0.621 s]
[INFO] flink-json ......................................... SUCCESS [ 14.060 s]
[INFO] flink-connector-kafka-base ......................... SUCCESS [ 12.890 s]
[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [  3.954 s]
[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [  4.675 s]
[INFO] flink-connector-elasticsearch-base ................. SUCCESS [ 18.038 s]
[INFO] flink-connector-elasticsearch2 ..................... SUCCESS [ 39.863 s]
[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [ 43.666 s]
[INFO] flink-connector-elasticsearch6 ..................... SUCCESS [ 19.390 s]
[INFO] flink-connector-elasticsearch7 ..................... SUCCESS [ 19.741 s]
[INFO] flink-orc .......................................... SUCCESS [ 12.435 s]
[INFO] flink-csv .......................................... SUCCESS [  4.265 s]
[INFO] flink-connector-hive ............................... SUCCESS [ 24.227 s]
[INFO] flink-connector-rabbitmq ........................... SUCCESS [  3.825 s]
[INFO] flink-connector-twitter ............................ SUCCESS [  5.428 s]
[INFO] flink-connector-nifi ............................... SUCCESS [  2.346 s]
[INFO] flink-connector-cassandra .......................... SUCCESS [ 29.089 s]
[INFO] flink-avro ......................................... SUCCESS [ 22.772 s]
[INFO] flink-connector-filesystem ......................... SUCCESS [ 13.748 s]
[INFO] flink-connector-kafka .............................. SUCCESS [  3.999 s]
[INFO] flink-connector-gcp-pubsub ......................... SUCCESS [  6.061 s]
[INFO] flink-connector-kinesis ............................ SUCCESS [ 26.639 s]
[INFO] flink-sql-connector-elasticsearch7 ................. SUCCESS [ 23.964 s]
[INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [ 21.835 s]
[INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [  1.523 s]
[INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [  2.891 s]
[INFO] flink-sql-connector-kafka .......................... SUCCESS [  2.267 s]
[INFO] flink-avro-confluent-registry ...................... SUCCESS [ 11.554 s]
[INFO] flink-parquet ...................................... SUCCESS [ 14.598 s]
[INFO] flink-sequence-file ................................ SUCCESS [  6.775 s]
[INFO] flink-compress ..................................... SUCCESS [  4.573 s]
[INFO] flink-examples-streaming ........................... SUCCESS [ 32.028 s]
[INFO] flink-examples-table ............................... SUCCESS [ 15.357 s]
[INFO] flink-examples-build-helper ........................ SUCCESS [  0.492 s]
[INFO] flink-examples-streaming-twitter ................... SUCCESS [  1.576 s]
[INFO] flink-examples-streaming-state-machine ............. SUCCESS [  1.231 s]
[INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [  8.301 s]
[INFO] flink-container .................................... SUCCESS [  5.040 s]
[INFO] flink-queryable-state-runtime ...................... SUCCESS [  9.197 s]
[INFO] flink-end-to-end-tests ............................. SUCCESS [  0.782 s]
[INFO] flink-cli-test ..................................... SUCCESS [  2.876 s]
[INFO] flink-parent-child-classloading-test-program ....... SUCCESS [  3.194 s]
[INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [  5.423 s]
[INFO] flink-dataset-allround-test ........................ SUCCESS [  3.042 s]
[INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [  3.037 s]
[INFO] flink-datastream-allround-test ..................... SUCCESS [  9.758 s]
[INFO] flink-batch-sql-test ............................... SUCCESS [  1.323 s]
[INFO] flink-stream-sql-test .............................. SUCCESS [  1.056 s]
[INFO] flink-bucketing-sink-test .......................... SUCCESS [  8.995 s]
[INFO] flink-distributed-cache-via-blob ................... SUCCESS [  2.875 s]
[INFO] flink-high-parallelism-iterations-test ............. SUCCESS [ 16.787 s]
[INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [ 12.093 s]
[INFO] flink-queryable-state-test ......................... SUCCESS [ 10.283 s]
[INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [  3.863 s]
[INFO] flink-elasticsearch2-test .......................... SUCCESS [  6.756 s]
[INFO] flink-elasticsearch5-test .......................... SUCCESS [  6.704 s]
[INFO] flink-elasticsearch6-test .......................... SUCCESS [ 15.619 s]
[INFO] flink-quickstart ................................... SUCCESS [  1.669 s]
[INFO] flink-quickstart-java .............................. SUCCESS [  4.340 s]
[INFO] flink-quickstart-scala ............................. SUCCESS [  4.757 s]
[INFO] flink-quickstart-test .............................. SUCCESS [  1.488 s]
[INFO] flink-confluent-schema-registry .................... SUCCESS [  4.117 s]
[INFO] flink-stream-state-ttl-test ........................ SUCCESS [ 20.106 s]
[INFO] flink-sql-client-test .............................. SUCCESS [  5.903 s]
[INFO] flink-streaming-file-sink-test ..................... SUCCESS [  2.908 s]
[INFO] flink-state-evolution-test ......................... SUCCESS [  7.972 s]
[INFO] flink-mesos ........................................ SUCCESS [01:39 min]
[INFO] flink-kubernetes ................................... SUCCESS [ 20.125 s]
[INFO] flink-yarn ......................................... SUCCESS [  6.253 s]
[INFO] flink-gelly ........................................ SUCCESS [ 42.014 s]
[INFO] flink-gelly-scala .................................. SUCCESS [ 36.521 s]
[INFO] flink-gelly-examples ............................... SUCCESS [ 27.928 s]
[INFO] flink-metrics-dropwizard ........................... SUCCESS [  2.265 s]
[INFO] flink-metrics-graphite ............................. SUCCESS [  0.839 s]
[INFO] flink-metrics-influxdb ............................. SUCCESS [  3.368 s]
[INFO] flink-metrics-prometheus ........................... SUCCESS [  2.525 s]
[INFO] flink-metrics-statsd ............................... SUCCESS [  1.714 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [  1.720 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [  1.715 s]
[INFO] flink-cep-scala .................................... SUCCESS [ 19.332 s]
[INFO] flink-table-uber ................................... SUCCESS [ 14.607 s]
[INFO] flink-table-uber-blink ............................. SUCCESS [  7.316 s]
[INFO] flink-sql-client ................................... SUCCESS [ 12.390 s]
[INFO] flink-state-processor-api .......................... SUCCESS [  5.691 s]
[INFO] flink-python ....................................... SUCCESS [ 32.964 s]
[INFO] flink-scala-shell .................................. SUCCESS [ 43.815 s]
[INFO] flink-dist ......................................... SUCCESS [ 22.283 s]
[INFO] flink-end-to-end-tests-common ...................... SUCCESS [  1.452 s]
[INFO] flink-metrics-availability-test .................... SUCCESS [  0.710 s]
[INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [  0.706 s]
[INFO] flink-heavy-deployment-stress-test ................. SUCCESS [ 30.449 s]
[INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [  5.651 s]
[INFO] flink-streaming-kafka-test-base .................... SUCCESS [  2.944 s]
[INFO] flink-streaming-kafka-test ......................... SUCCESS [ 14.949 s]
[INFO] flink-streaming-kafka011-test ...................... SUCCESS [ 15.905 s]
[INFO] flink-streaming-kafka010-test ...................... SUCCESS [ 12.944 s]
[INFO] flink-plugins-test ................................. SUCCESS [  2.659 s]
[INFO] dummy-fs ........................................... SUCCESS [  1.475 s]
[INFO] another-dummy-fs ................................... SUCCESS [  1.371 s]
[INFO] flink-tpch-test .................................... SUCCESS [  9.452 s]
[INFO] flink-streaming-kinesis-test ....................... SUCCESS [ 29.559 s]
[INFO] flink-elasticsearch7-test .......................... SUCCESS [ 15.898 s]
[INFO] flink-end-to-end-tests-common-kafka ................ SUCCESS [  2.041 s]
[INFO] flink-tpcds-test ................................... SUCCESS [  8.540 s]
[INFO] flink-statebackend-heap-spillable .................. SUCCESS [  2.477 s]
[INFO] flink-contrib ...................................... SUCCESS [  0.504 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [  3.773 s]
[INFO] flink-yarn-tests ................................... SUCCESS [  5.609 s]
[INFO] flink-fs-tests ..................................... SUCCESS [  6.487 s]
[INFO] flink-docs ......................................... SUCCESS [  4.947 s]
[INFO] flink-ml-parent .................................... SUCCESS [  0.971 s]
[INFO] flink-ml-api ....................................... SUCCESS [  1.593 s]
[INFO] flink-ml-lib ....................................... SUCCESS [ 11.757 s]
[INFO] flink-walkthroughs ................................. SUCCESS [  1.885 s]
[INFO] flink-walkthrough-common ........................... SUCCESS [  8.696 s]
[INFO] flink-walkthrough-table-java ....................... SUCCESS [  4.048 s]
[INFO] flink-walkthrough-table-scala ...................... SUCCESS [  4.153 s]
[INFO] flink-walkthrough-datastream-java .................. SUCCESS [  4.061 s]
[INFO] flink-walkthrough-datastream-scala ................. SUCCESS [  5.361 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:33 min (Wall Clock)
[INFO] Finished at: 2020-01-15T17:03:40+08:00
[INFO] Final Memory: 584M/1829M
[INFO] ------------------------------------------------------------------------

切换到flink-dist/target/flink-1.11-SNAPSHOT-bin目录，将生成后的文件打包成安装包：

tar -czvf flink-1.11-cdh-6.1.0.tar.gz flink-1.11-SNAPSHOT/

上传到服务器并使用。

猫不夜行
关注
  • 2
    点赞
  • 踩
  • 7
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
专栏目录
CDH6.3.2编译并集成Flink
等闲之辈莫等闲的博客
06-10 3182
一、安装包准备 CDH 6.3.2(Hadoop 3.0.0)、Flink 1.12.4、flink-shaded-12.0、Centos7.9、Maven 3.6.3、Scala-2.13.6和JDK1.8.0_291 flink包下载：https://mirrors.tuna.tsinghua.edu.cn/apache/flink/ 二、环境配置 1.将安装包上传到相应目录并解压 tar -xzvf xx.tgz 2.配置环境变量 vi /etc/profile export JAVA_HOME=/
CDH6配置Flink(编译安装)
qq_31886877的博客
09-09 1851
一、环境 CDH 6.1.1(Hadoop 3.0.0)、Flink 1.10.1、Centos7.4、Maven3.6.3、Scala-2.12和CDH的原生jdk1.8 二、安装包 1.flink1.10.1 tar包 https://archive.apache.org/dist/flink/flink-1.10.1/flink-1.10.1-src.tgz 其他版本可以通过该方法下载资源 [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-2WuVWRyy-15996210
CDH6.2.0（别的也适用） 编译flink1.14.3
尤托匹亚的博客
04-05 2579
1. 方法有两种 1.1 克隆cloudera提供的打包，然后下载flink一步一步按照教程来，比较麻烦。 1.2 采用大佬开发好的脚本，直接一个shell搞定。（最好搞过javaEE和看得懂shell） 1.3 免费下载我打包好的程序 CDH-6.2-FLINK_ON_YARN-1.14.3.jar CDH-6.2-FLINK-1.14.3-BIN-SCALA_2.12.tar 2. 白剽大佬的项目 flink-parcel 3. 江湖规矩，有图为证 ...
CDH6.3.2编译Flink-1.12.4
04-12
具体编译和集成过程，以及其他组件版本请参考https://blog.csdn.net/weixin_56131938/article/details/117778364
基于3.0.0-cdh6.3.2版本编译Flink1.14.4
永青技术博客
05-25 1300
java.lang.Runtimelxception:java,lang.IllegalArgumentException:UnrecoonizedHadoopmajorversionnumber:3.0.0-cdh6.2.1
CDH-6.3.2-FLINK1.12.0相关资源包
03-15
这个"CDH-6.3.2-FLINK1.12.0相关资源包"便是为了帮助用户在CDH平台上顺利部署和运行Flink应用而设计的。 Flink 1.12.0是Flink社区的一个稳定版本，它引入了许多新特性，例如改进了SQL支持，提供了更好的状态管理和...
flink-connector-kudu:基于Apache-bahir-kudu-connector的flink-connector-kudu，支持Flink1.11.x DynamicTableSourceSink，支持范围分区等
03-04
基于Apache-Bahir-Kudu连接器改造而来的满足公司内部使用的Kudu连接器，支持特性范围分区，定义哈希分桶数，支持Flink1.11.x动态数据源等，改造后已贡献部分功能给社区。 使用姿势 clone代码后，改造pom项目坐标后...
hive-jdbc-2.1.1-cdh6.1.0-standalone.jar
03-16
hive JDBC jar包。由于项目使用，此jar包从国外下载费了好大劲，现分享给大家。 hive JDBC jar包。由于项目使用，此jar包从国外下载费了好大劲，现分享给大家。 hive JDBC jar包。由于项目使用，此jar包从国外下载费...
flink-1.12-CDH-6.3.2.zip
03-19
标题“flink-1.12-CDH-6.3.2.zip”指的是一个包含Apache Flink 1.12版本与Cloudera Data Hub (CDH) 6.3.2集成的压缩文件。这个文件是为在CDH环境中运行Flink作业而准备的，确保了兼容性和优化。 描述“flink1.12...
基于CDH-5.14.2编译Flink1.9.1的安装包
猫不夜行的博客
01-03 3772
准备工作 maven3 jdk1.8 CDH-5.14.2 下载源码包 [root@node01 ~]# cd /opt/software/ [root@node01 software]# wget http://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-src.tgz [root@node01 software]# t...
Flink在CDH配置部署及读取kafka的实例验证；
05-06
CDH目前不带flink组件，文档为Flink在CDH集群上的配置部署及读取kafka的实例验证；
Flink1.11.0源码手动编译包
07-25
Flink1.11.0源码手动编译包 编译步骤详见：https://blog.csdn.net/dzh284616172/article/details/107571972
cdh.6.3.2 编译成功的flink1.10.1
11-24
CDH6x最后一个免费版本下可以使用的Flink，最新编译成功的flink1.10.1版本，可以直接使用，编译适用环境CDH6.3.2版本。
flink-1.13.1 cdh6.3.2
07-24
flink 部署在cdh6.3.2 的包
FLINK-1.14.0 cdh6.3.2 .zip
10-06
flink 部署在cdh6.3.2 的包
CDH6 配置Flink1.10(编译安装)
我思，顾我在
06-19 3273
以下包含了CDH6整合Flink1.10编译安装的全过程以及趟过的坑，供大家参考、避坑。 一、环境 CDH 6.2.1(Hadoop 3.0.0)、Flink 1.10.1、Centos7.7、Maven 3.6.3、Scala-2.12和JDK1.8 二、安装包 1、flink1.10.1 tar包 https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.10.1/flink-1.1...
cdh5.14.2版flink1.10编译过程
小5哥的专栏
05-25 555
一、编译环境 1、Java（至少1.8以上） #java -version openjdk version "1.8.0_252" OpenJDK Runtime Environment (build 1.8.0_252-b09) OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode) 2、Maven(至少Maven 3以上版本） # mvn -v Apache Maven 3.6.3 (cecedd343002696d0abb50b32...
flink集成cdh编译
记录工作所遇问题及个人概念理解
10-11 577
一、环境准备 1、环境：Jdk 1.8、centos7.6、Maven 3.6.3和Scala-2.12 2、源码和CDH 版本：Flink 1.9.0 、 CDH 6.1.1(Hadoop 3.0.0) 注：mvn版本、CDH版本和scala版本无所谓，理论上可根据自己的版本自行更改。 二、安装包准备； 1、maven版本不要太低，我用的是maven 3.6.3： https://mirror.bit.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-ma
CDH6.3.2集成安装flink on yarn服务（编译源码，制作parcel）（更新：flink1.12.2版本测试也可用）
qq_31454379的博客
12-01 7185
目录一：环境准备二：下载安装包1. Flink-shaded包2. flink1.10.2 源码包3. maven配置文件三：编译flink-shaded版本1. 解压tar包2. 解压文件后目录结构3. 修改pom文件4. 开始编译四：编译Flink1.10.2源码1. 解压tar包2. 执行编译3. 等待编译成功4. 打包编译好的文件五：制作parcel包1. 下载git开源制作parcel包的项目2. 修改配置文件 flink-parcel.properties3. 赋予build.sh文件执行权限4
zookeeper-3.4.5-cdh6.1.0.tar.gz
最新发布
09-13
zookeeper-3.4.5-cdh6.1.0.tar.gz是Cloudera发行的一个特定版本的ZooKeeper软件包。根据提供的引用内容和，我们可以看到这个版本的ZooKeeper与其他大数据组件存在一些差异。具体来说，引用中列出的ZooKeeper版本是...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值