Iceberg0.11与Spark3.0结合

Iceberg0.11与Spark3.0结合

1.安装编译Iceberg0.11

此处我下载的是Iceberg0.11.1版本
需要提前安装gradle,iceberg的编译，此处使用的是gradle5.4.1版本

wget https://downloads.gradle.org/distributions/gradle-5.4-bin.zip
unzip -d /opt/gradle gradle-5.4-bin.zip
vim /etc/profile
# 加入下面的
#GREDLE
export GRADLE_HOME=/opt/gradle/gradle-5.4
export PATH=$PATH:$GRADLE_HOME/bin

source /etc/profile

编译Iceberg，github上下载源码进行编译，此处略过下载过程直接进行编译

cd iceberg-apache-iceberg-0.11.1
gradle build -x test

2. Iceberg编译与SparkSQL相结合

2.1上述编译成功后到spark3目录下取出我们所需的jar包

cd spark3-runtime/build/libs
ll

iceberg-spark3-runtime-0.11.1.jar为我们所需的插件包

2.2 将插件包放入到spark目录下

cd $SPARK_HOME/jars
cp iceberg-apache-iceberg-0.11.1/spark3-runtime/build/libs/iceberg-spark3-runtime-0.11.1.jar .