实践：DeltaLake-0.8.0与Spark-3.1.1编译与在cdh6.3.2上应用

前言

一次Delta lake 0.8.0 踩坑有感：使用新框架的新版本，一定要尽早关注多多关注社区动态

实践

环境准备

1. 安装sbt，参考：centos 安装sbt

[root@master soft]# yum install sbt
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
 * base: mirrors.ustc.edu.cn
 * extras: mirrors.ustc.edu.cn
 * updates: mirrors.ustc.edu.cn
base                                                                                                                 | 3.6 kB  00:00:00
extras                                                                                                               | 2.9 kB  00:00:00
updates                                                                                                              | 2.9 kB  00:00:00
No package sbt available.
Error: Nothing to do
[root@master soft]# curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   160    0   160    0     0    111      0 --:--:--  0:00:01 --:--:--   111
[root@master soft]# cp bintray-sbt-rpm.repo /etc/yum.repos.d/
[root@master soft]# yum install sbt

2. 下载delta源码并编译，参考：Delta File Fomat 1:环境搭建和测试

[root@master soft]# git clone https://github.com/delta-io/delta.git
[root@master soft]# cd delta/
[root@master delta]# build/sbt compile package