写在前头:最近在CDH上集成flink,现在新版本的CDH6.3以上的,官网有提供parcel跟csd文件,但是我们使用的是CDH5.12.1,还是比较老的版本,所以要自己编译。在这期间搜集了不少资料,但是没有一篇详细的文章是从flink编译=>生成csd、parcel文件=>集成cm这一整套流程完整走下来的。因此我整理了之前收集的资料,并把自己在安装过程出现的问题也整理了出来。希望对大家有所帮助
一、安装准备
我使用的软件版本:
Hadoop 2.6.0-cdh5.12.1
Apache Maven 3.6.3
Java version: 1.8.0_161
Cloudera Manager 5.12.1
我们需要做的事情:
1、从官网上下载flink的源码及依赖包
2、对源码及依赖包进行相应的编译
3、将编译完成的flink包制作成cloudera manager需要的csd文件、parcel文件
4、cm一键安装部署flink,并将flink服务添加到管理列表中
明确了我们需要做的事情之后,一步步操作就好了,这个过程我会尽可能详细的描述操作过程。
二、软件包下载
这里可以根据自己需要的版本下载自己需要包即可。这里我选择的1.9.0,对应的依赖包是7.0
- flink的安装包下载
wget https://github.com/apache/flink/archive/release-1.9.0.zip
- flink依赖包
wget https://archive.apache.org/dist/flink/flink-shaded-7.0/flink-shaded-7.0-src.tgz
三、编译
- flink-1.9.0编译
1.解压
unzip release-1.9.0.zip -d ./
2.修改根目录下的pom.xml
vim flink-release-1.9.0/pom.xml
删除test相关modules,减少编译时间,大约在80-90行
增加下载源
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>mvnrepository</id>
<url>https://mvnrepository.com</url>
</repository>
</repositories>
3.开始编译
mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.12.1 -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true
编译成功
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for flink 1.9.0:
[INFO]
[INFO] force-shading ...................................... SUCCESS [ 1.579 s]
[INFO] flink .............................................. SUCCESS [ 1.484 s]
[INFO] flink-annotations .................................. SUCCESS [ 40.863 s]
[INFO] flink-shaded-curator ............................... SUCCESS [01:34 min]
[INFO] flink-metrics ...................................... SUCCESS [ 0.059 s]
[INFO] flink-metrics-core ................................. SUCCESS [01:58 min]
[INFO] flink-test-utils-parent ............................ SUCCESS [ 0.066 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [ 0.485 s]
[INFO] flink-core ......................................... SUCCESS [02:14 min]
[INFO] flink-java ......................................... SUCCESS [ 19.734 s]
[INFO] flink-queryable-state .............................. SUCCESS [ 0.047 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [ 0.469 s]
[INFO] flink-filesystems .................................. SUCCESS [ 0.050 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [04:23 min]
[INFO] flink-runtime ...................................... SUCCESS [32:45 min]
[INFO] flink-scala ........................................ SUCCESS [01:01 min]
[INFO] flink-mapr-fs ...................................... SUCCESS [ 0.397 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [02:05 min]
[INFO] flink-s3-fs-base ................................... SUCCESS [01:35 min]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [ 18.102 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [08:16 min]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [03:06 min]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [02:38 min]
[INFO] flink-azure-fs-hadoop .............................. SUCCESS [12:05 min]
[INFO] flink-optimizer .................................... SUCCESS [ 1.839 s]
[INFO] flink-clients ...................................... SUCCESS [ 9.584 s]
[INFO] flink-streaming-java ............................... SUCCESS [ 12.332 s]
[INFO]