flink 1.11.2 编译 hadoop3.0 CDH6.3

最新推荐文章于 2023-02-17 16:03:30 发布
置顶 最新推荐文章于 2023-02-17 16:03:30 发布
阅读量1.4k 收藏 1
点赞数
文章标签： flink 1.11 flink sql
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/YAJUN0601/article/details/109235026
版权

环境准备：

1. maven 安装、修改阿里源

 

获取源码

git clone https://github.com/apache/flink

git tags

git checkout release-1.11.2  # 最新的tag

开始编译

cd flink

# CDH6.3 使用的 hadoop 版本是 3.0.0

mvn clean package -DskipTests  -Dhadoop.version=3.0.0

 

遇到错误：
 

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12:47 min
[INFO] Finished at: 2020-10-23T09:10:19+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-azure-fs-hadoop: Could not resolve dependencies for project org.apache.flink:flink-azure-fs-hadoop:jar:1.11.2: The following artifacts could not be resolved: com.nimbusds:lang-tag:jar:1.5, com.nimbusds:nimbus-jose-jwt:jar:9.1: Could not find artifact com.nimbusds:lang-tag:jar:1.5 -> [Help 1]
[ERROR] 

这个包的作用：Java implementation of "Tags for Identifying Languages", RFC-5646.

解决方案：百度找到这个包，pom 文件中添加：

<!-- https://www.KuMaPai.com/open/com.nimbusds/lang-tag -->
<dependency>
<groupId>com.nimbusds</groupId>
<artifactId>lang-tag</artifactId>
<version>1.5</version>
</dependency>

重新运行编译

又遇到错误：

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.865 s
[INFO] Finished at: 2020-10-23T10:17:20+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-azure-fs-hadoop: Could not resolve dependencies for project org.apache.flink:flink-azure-fs-hadoop:jar:1.11.2: The following artifacts could not be resolved: org.apache.flink:flink-fs-hadoop-shaded:jar:1.11.2, com.nimbusds:nimbus-jose-jwt:jar:9.1: Failure to find org.apache.flink:flink-fs-hadoop-shaded:jar:1.11.2 in https://maven.aliyun.com/repository/central was cached in the local repository, resolution will not be reattempted until the update interval of alimaven has elapsed or updates are forced -> [Help 1]

pom 文件查看 依赖这个 <flink.shaded.version>11.0</flink.shaded.version>

 

下载了两个版本的 7.0 和 11.0，分别编译安装才解决问题

发现11.0 有几个包不存在

mvn clean install -DskipTests -Dhadoop.version=2.7.7

然后又报错，找不到 kafka-schema-registry-client/4.1.0

自己从网上下载：http://packages.confluent.io/maven/io/confluent/kafka-schema-registry-client/4.1.0/

cp kafka-schema-registry-client-4.1.0.jar /data/.m2/repository/io/confluent/kafka-schema-registry-client/4.1.0/

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] flink-avro-confluent-registry 1.11.2 ............... SUCCESS [ 24.706 s]
[INFO] flink-sequence-file 1.11.2 ......................... SUCCESS [ 26.166 s]
[INFO] flink-compress 1.11.2 .............................. SUCCESS [ 26.121 s]
[INFO] flink-sql-orc 1.11.2 ............................... SUCCESS [ 24.413 s]
[INFO] flink-sql-parquet 1.11.2 ........................... SUCCESS [  4.973 s]
[INFO] flink-examples-streaming 1.11.2 .................... SUCCESS [01:20 min]
[INFO] flink-examples-table 1.11.2 ........................ SUCCESS [ 43.440 s]
[INFO] flink-examples-build-helper 1.11.2 ................. SUCCESS [  1.798 s]
[INFO] flink-examples-streaming-twitter 1.11.2 ............ SUCCESS [  1.174 s]
[INFO] flink-examples-streaming-state-machine 1.11.2 ...... SUCCESS [  1.376 s]
[INFO] flink-examples-streaming-gcp-pubsub 1.11.2 ......... SUCCESS [ 14.383 s]
[INFO] flink-container 1.11.2 ............................. SUCCESS [  4.036 s]
[INFO] flink-queryable-state-runtime 1.11.2 ............... SUCCESS [  6.607 s]
[INFO] flink-mesos 1.11.2 ................................. SUCCESS [01:37 min]
[INFO] flink-kubernetes 1.11.2 ............................ SUCCESS [ 34.664 s]
[INFO] flink-yarn 1.11.2 .................................. SUCCESS [ 10.983 s]
[INFO] flink-gelly 1.11.2 ................................. SUCCESS [ 16.968 s]
[INFO] flink-gelly-scala 1.11.2 ........................... SUCCESS [ 48.855 s]
[INFO] flink-gelly-examples 1.11.2 ........................ SUCCESS [ 18.459 s]
[INFO] flink-external-resources 1.11.2 .................... SUCCESS [  0.928 s]
[INFO] flink-external-resource-gpu 1.11.2 ................. SUCCESS [  1.286 s]
[INFO] flink-metrics-dropwizard 1.11.2 .................... SUCCESS [  0.960 s]
[INFO] flink-metrics-graphite 1.11.2 ...................... SUCCESS [  0.479 s]
[INFO] flink-metrics-influxdb 1.11.2 ...................... SUCCESS [  3.117 s]
[INFO] flink-metrics-prometheus 1.11.2 .................... SUCCESS [  0.930 s]
[INFO] flink-metrics-statsd 1.11.2 ........................ SUCCESS [  0.567 s]
[INFO] flink-metrics-datadog 1.11.2 ....................... SUCCESS [  0.476 s]
[INFO] flink-metrics-slf4j 1.11.2 ......................... SUCCESS [  0.604 s]
[INFO] flink-cep-scala 1.11.2 ............................. SUCCESS [ 49.727 s]
[INFO] flink-table-uber 1.11.2 ............................ SUCCESS [ 23.841 s]
[INFO] flink-table-uber-blink 1.11.2 ...................... SUCCESS [ 25.715 s]
[INFO] flink-python 1.11.2 ................................ SUCCESS [ 36.904 s]
[INFO] flink-sql-client 1.11.2 ............................ SUCCESS [ 36.687 s]
[INFO] flink-state-processor-api 1.11.2 ................... SUCCESS [  3.764 s]
[INFO] flink-ml-parent 1.11.2 ............................. SUCCESS [  0.487 s]
[INFO] flink-ml-api 1.11.2 ................................ SUCCESS [  1.240 s]
[INFO] flink-ml-lib 1.11.2 ................................ SUCCESS [  4.884 s]
[INFO] flink-ml-uber 1.11.2 ............................... SUCCESS [  0.561 s]
[INFO] flink-scala-shell 1.11.2 ........................... SUCCESS [ 37.075 s]
[INFO] flink-dist 1.11.2 .................................. SUCCESS [ 39.390 s]
[INFO] flink-yarn-tests 1.11.2 ............................ SUCCESS [ 17.936 s]
[INFO] flink-end-to-end-tests 1.11.2 ...................... SUCCESS [ 31.051 s]
[INFO] flink-cli-test 1.11.2 .............................. SUCCESS [  0.751 s]
[INFO] flink-parent-child-classloading-test-program 1.11.2  SUCCESS [  0.729 s]
[INFO] flink-parent-child-classloading-test-lib-package 1.11.2 SUCCESS [  0.656 s]
[INFO] flink-dataset-allround-test 1.11.2 ................. SUCCESS [  0.810 s]
[INFO] flink-dataset-fine-grained-recovery-test 1.11.2 .... SUCCESS [  0.999 s]
[INFO] flink-datastream-allround-test 1.11.2 .............. SUCCESS [ 11.424 s]
[INFO] flink-batch-sql-test 1.11.2 ........................ SUCCESS [  1.152 s]
[INFO] flink-stream-sql-test 1.11.2 ....................... SUCCESS [  1.606 s]
[INFO] flink-bucketing-sink-test 1.11.2 ................... SUCCESS [  3.264 s]
[INFO] flink-distributed-cache-via-blob 1.11.2 ............ SUCCESS [  1.694 s]
[INFO] flink-high-parallelism-iterations-test 1.11.2 ...... SUCCESS [ 20.312 s]
[INFO] flink-stream-stateful-job-upgrade-test 1.11.2 ...... SUCCESS [  2.027 s]
[INFO] flink-queryable-state-test 1.11.2 .................. SUCCESS [  9.986 s]
[INFO] flink-local-recovery-and-allocation-test 1.11.2 .... SUCCESS [  3.593 s]
[INFO] flink-elasticsearch5-test 1.11.2 ................... SUCCESS [ 21.726 s]
[INFO] flink-elasticsearch6-test 1.11.2 ................... SUCCESS [ 14.844 s]
[INFO] flink-quickstart 1.11.2 ............................ SUCCESS [  2.979 s]
[INFO] flink-quickstart-java 1.11.2 ....................... SUCCESS [  1.682 s]
[INFO] flink-quickstart-scala 1.11.2 ...................... SUCCESS [  0.996 s]
[INFO] flink-quickstart-test 1.11.2 ....................... SUCCESS [ 10.481 s]
[INFO] flink-confluent-schema-registry 1.11.2 ............. SUCCESS [  4.667 s]
[INFO] flink-stream-state-ttl-test 1.11.2 ................. SUCCESS [  9.904 s]
[INFO] flink-sql-client-test 1.11.2 ....................... SUCCESS [ 33.799 s]
[INFO] flink-streaming-file-sink-test 1.11.2 .............. SUCCESS [  0.517 s]
[INFO] flink-state-evolution-test 1.11.2 .................. SUCCESS [  1.934 s]
[INFO] flink-rocksdb-state-memory-control-test 1.11.2 ..... SUCCESS [  1.719 s]
[INFO] flink-end-to-end-tests-common 1.11.2 ............... SUCCESS [  1.707 s]
[INFO] flink-metrics-availability-test 1.11.2 ............. SUCCESS [  0.535 s]
[INFO] flink-metrics-reporter-prometheus-test 1.11.2 ...... SUCCESS [  0.576 s]
[INFO] flink-heavy-deployment-stress-test 1.11.2 .......... SUCCESS [ 19.433 s]
[INFO] flink-connector-gcp-pubsub-emulator-tests 1.11.2 ... SUCCESS [ 29.240 s]
[INFO] flink-streaming-kafka-test-base 1.11.2 ............. SUCCESS [  0.529 s]
[INFO] flink-streaming-kafka-test 1.11.2 .................. SUCCESS [ 18.715 s]
[INFO] flink-streaming-kafka011-test 1.11.2 ............... SUCCESS [ 16.210 s]
[INFO] flink-streaming-kafka010-test 1.11.2 ............... SUCCESS [ 11.300 s]
[INFO] flink-plugins-test 1.11.2 .......................... SUCCESS [  0.228 s]
[INFO] dummy-fs 1.11.2 .................................... SUCCESS [  0.356 s]
[INFO] another-dummy-fs 1.11.2 ............................ SUCCESS [  0.808 s]
[INFO] flink-tpch-test 1.11.2 ............................. SUCCESS [  1.570 s]
[INFO] flink-streaming-kinesis-test 1.11.2 ................ SUCCESS [ 21.545 s]
[INFO] flink-elasticsearch7-test 1.11.2 ................... SUCCESS [ 47.795 s]
[INFO] flink-end-to-end-tests-common-kafka 1.11.2 ......... SUCCESS [  1.659 s]
[INFO] flink-tpcds-test 1.11.2 ............................ SUCCESS [  1.460 s]
[INFO] flink-netty-shuffle-memory-control-test 1.11.2 ..... SUCCESS [  0.309 s]
[INFO] flink-python-test 1.11.2 ........................... SUCCESS [ 25.629 s]
[INFO] flink-statebackend-heap-spillable 1.11.2 ........... SUCCESS [  1.679 s]
[INFO] flink-contrib 1.11.2 ............................... SUCCESS [  0.206 s]
[INFO] flink-connector-wikiedits 1.11.2 ................... SUCCESS [  0.989 s]
[INFO] flink-fs-tests 1.11.2 .............................. SUCCESS [  2.487 s]
[INFO] flink-docs 1.11.2 .................................. SUCCESS [  4.516 s]
[INFO] flink-walkthroughs 1.11.2 .......................... SUCCESS [  0.337 s]
[INFO] flink-walkthrough-common 1.11.2 .................... SUCCESS [  1.751 s]
[INFO] flink-walkthrough-datastream-java 1.11.2 ........... SUCCESS [  0.577 s]
[INFO] flink-walkthrough-datastream-scala 1.11.2 .......... SUCCESS [  0.455 s]
[INFO] flink-shaded-force-shading 11.0 .................... SUCCESS [  1.885 s]
[INFO] flink-shaded-zookeeper-3.5 3.5.6-11.0 .............. SUCCESS [  8.530 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:44 min (Wall Clock)

 

远涉江湖
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
flink 1.9.0 编译：flink-fs-hadoop-shaded 找不到
九师兄
09-24 29万+
1. 编译 lcc@lcc flink$ mvn clean install -Dmaven.test.skip=true -Dhadoop.version=2.8.3 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true 报错 [INFO] -----------------------------------------------------...
flink 1.10/1.11 master 编译报错：flink-azure-fs-hadoop 找不到json-smart等
X_Yde
12-26 1968
编译flink时，报如下类似错误： [ERROR] Failed to execute goal on project flink-connector-gcp-pubsub_2.11: Could not resolve dependencies for project org.apache.flink:flink-connector-gcp-pubsub_2.11:jar:1.10-SN...
flink-1.9.3-hadoop3.0.0-cdh6.2.1.tar.gz
06-10
资源是flink的编译包， Flink 编译和运行要求 Java 版本至少是 Java 8，且最好选用 Java 8u51 及以上版本，我选用的是java version "1.8.0_181"，源码是基于scala2.11，maven是3.2.5
CDH6.2/6.3 集成安装Flink 简明教程
勇敢的俍心的博客
06-03 6982
CDH6系列并不自带Flink，目前此类教程比较少，特此分享出来一个最简单的集成方式。 第一步：下载需要的jar，parcel等文件 wget https://archive.cloudera.com/csa/1.0.0.0/csd/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0.jar wget https://archive.cloudera.com/csa/1.0.0.0/parcels/manifest.json wget https://archive.cloudera.co
Flink on yarn 不废话集群部署
weixin_42507474的博客
10-28 2154
文章目录Flink on yarn 集群部署前言先安装好yarn集群，在我们这个环境中使用的是CDH6.3，也就是基于hadoop3.0的大数据生态环境flink部署1.包下载2.将所下载的包放置/opt/flink下进行解压安装3.切换到相关目录4.配置hadoop相关路径5.如果第四步采用后仍然报错找不到相关包，则切换到lib包中并将相关hadoop包放置lib目录下6.确保你的环境有足够的内存能够启动集群后，开始进行启动7.启动成功后，会提供客户端网址，输入浏览器可进行查看，成功代表集群已经启动 Fl
【Flink】Flink的编译（包含hadoop的依赖）
lsr40的博客
08-01 5429
好久没写文章了，手都有点生。 菜鸡一只，如果有说错的还请大家批评！ 最近工作上的事情还是有点忙的，主要都是一些杂活，不干又不行，干了好像提升又不多，不过拿人家手短吃人家嘴软，既然拿了工资就应该好好的干活，当然前提是需求相对合理的情况嘿嘿~ 近来Flink的势头有点猛啊，它和spark的区别在于：spark更倾向于批处理或者微批处理（spark现在的发展方向往人工智能的分布式算法上走了），但是...
CDH 6.3.2下安装Flink
weixin_47444415的博客
02-17 1500
cdh 6.3.2下安装flink简单教程，自己在生产环境中测试成功
CDH6.3.2集成Apache Flink1.12.0
终回首的博客
11-01 2320
目录一、编译Flink1 下载flink源码2 增加maven镜像3 执行编译命令二、编译parcel1 下载flink-parcel2 修改参数3 复制安装包4 编译parcel5 编译csd6 上传文件三、CDH集成1 登录CDH2 进入Parcel操作界面3 分配Parcel4 激活Parcel5 回主界面6 添加服务报错解决参考资料 版本： Centos7.6 JDK1.8 Scala2.11 Python2.7 Git1.8.3.1 Apache Maven3.6.3 CDH6.3.2 Apach
Failed to execute goal on project hadoop: Could not resolve dependencies for project com.huadian.big
IT_private的博客
08-23 1787
mvn package 打包的时候如果是上面这个错误：就是少了个jar包 就是这个jar包。下载网址： https://public.nexus.pentaho.org/content/groups/omni/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/ 放到这个路径下就行了。...
cdh6.3.2集成flink1.10.2-parcel包(亲测可用)
12-01
源码编译制作的parcel包，centos7.5+cdh6.3.2+scala2.12亲测可用，内附文章教程地址。
flink-shaded-hadoop-2-uber-3.0.0-cdh6.2.0-7.0.jar（jar包）.rar
03-07
Flink jar包，官网下载很慢，有需要的自行下载 Apache Flink是由Apache软件基金会开发的开源流处理框架，其核心是用Java和Scala编写的分布式流数据流引擎。Flink以数据并行和流水线方式执行任意流数据程序，Flink的流水线运行时系统可以执行批处理和流处理程序。此外，Flink的运行时本身也支持迭代算法的执行。
CDH6.3.2集成Flink1.12.2-parcel包(apache Flink-1.12.2-bin-scala_2.12.tgz)
04-15
源码编译制作的parcel包，linux+cdh6.3.2+scala2.12必然可用，内附文章的教程地址。
flink-1.10.1-hadoop-3.0.0-cdh6.2.1.tar.gz
06-10
资源是flink的编译包， Flink 编译和运行要求 Java 版本至少是 Java 8，且最好选用 Java 8u51 及以上版本，我选用的是java version "1.8.0_181"，源码是基于scala2.11，maven是3.2.5
flink-1.12-CDH-6.3.2.zip
03-19
flink1.12集成CDH6.3.2，亲测可用
flink 1.12及以上 兼容cdh6所需要的flink-shaded-hadoop jar包
02-11
flink 1.12及以上 兼容cdh6所需要的flink-shaded-hadoop jar包
Flink1.12.2-CDH6.3.2.7z
11-25
基于CDH6.3.2 集成的Flink1.12.2,亲测可用
flink-1.11.2-src.tgz
10-28
flink-1.11.2-src.tgz 官网下载比较慢的，可以到csdn这里下载哦.........
cdh.6.3.2 编译成功的flink1.10.1
11-24
CDH6x最后一个免费版本下可以使用的Flink，最新编译成功的flink1.10.1版本，可以直接使用，编译适用环境CDH6.3.2版本。
hadoop1.0和hadoop3.0
最新发布
10-14
Hadoop是一个开源的分布式计算框架，...其中最重要的改变是引入了YARN（Yet Another Resource Negotiator），它可以让Hadoop支持更多的计算框架，例如Spark、Flink等。此外，Hadoop 3.0还支持容器化、多租户等功能。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值