iceberg0.12 master集成flink 1.13.3,自己下载master源码打包
1. flink1.13.3下载地址
1. flink1.13.3
wget https://dlcdn.apache.org/flink/flink-1.13.3/flink-1.13.3-bin-scala_2.12.tgz
这个包放到flink lib下
wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-2.3.6_2.11/1.12.2/flink-sql-connector-hive-2.3.6_2.11-1.12.4.jar
2. iceberg0.12 master自己打包
登录linux
3. 安装git
yum -y install git
4. 下载iceberg代码
git clone https://github.com/apache/iceberg.git
cd iceberg
5. 修改 gradlew 仓库地址,添加阿里云仓库
vim build.properties
添加阿里云地址
maven { url 'https://maven.aliyun.com/repository/public/' }
添加后的完成配置:
buildscript {
repositories {
maven { url 'https://maven.aliyun.com/repository/public/' }
gradlePluginPortal()
}
dependencies {
classpath 'gradle.plugin.com.github.jengelman.gradle.plugins:shadow:7.0.0'
classpath 'com.palantir.baseline:gradle-baseline-java:4.0.0'
classpath 'com.palantir.gradle.gitversion:gradle-git-version:0.12.3'
classpath 'com.diffplug.spotless:spotless-plugin-gradle:5.14.1'
classpath 'gradle.plugin.org.inferred:gradle-processors:3.3.0'
classpath 'me.champeau.gradle:jmh-gradle-plugin:0.5.3'
classpath "com.github.alisiikh:gradle-scalastyle-plugin:3.4.1"
}
}
6. gradlew 打包
./gradlew build -x test
等待打包,第一次会下载相关依赖,会很久
7. 打包后拿到两个包
一个iceberg-flink-runtime*.jar 放到flink lib目录下
一个iceberg-hive-runtime*.jar 放到hive auxlib目录下
卸载scala2.11 安装scala2.12
yum remove scala
wget -O scala-2.12.6.rpm https://downloads.lightbend.com/scala/2.12.6/scala-2.12.6.rpm
rpm -rv scala-2.12.6.rpm