使用Java/Scala API对Iceberg数据湖进行snapshot快照expire过期处理

1. expire过期处理的作用

expire过期的snapshot-id，就不会出现在新的vN.metadata.json文件中

2. 查看表snapshot-log

查看最新版本的vN.metadata.json，显示的snapshot-log部分如下：

"snapshot-log" : [ {
    "timestamp-ms" : 1645089884916,
    "snapshot-id" : 3105878303282846379
  }, {
    "timestamp-ms" : 1645089898911,
    "snapshot-id" : 781839495286765092
  }, {
    "timestamp-ms" : 1645089917661,
    "snapshot-id" : 242516093407225541
  }, {
    "timestamp-ms" : 1645089922843,
    "snapshot-id" : 6992061996353419515
  }, {
    "timestamp-ms" : 1645090105742,
    "snapshot-id" : 4375332003125072899
  }, {
    "timestamp-ms" : 1645090219342,
    "snapshot-id" : 4792595715782867813
  } ]

3. expire过期程序

3.1 pom.xml

添加的依赖部分如下：

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.3.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs-client</artifactId>
            <version>3.3.1</version>
        </dependency>
        
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.10.1</version>
        </dependency>
        
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-core</artifactId>
            <version>0.13.0</version>
        </dependency>

3.2 放置hdfs-site.xml

将hdfs-site.xml放到项目的resources目录下

3.3 flink_test.scala

import org.apache.hadoop.conf.Configuration
import org.apache.iceberg.Table
import org.apache.iceberg.catalog.TableIdentifier
import org.apache.iceberg.hadoop.HadoopCatalog


object flink_test {

  def main(args: Array[String]): Unit = {

    // =======================expire过期快照========================
    val icebergWarehousePath = "hdfs://nnha/user/iceberg/warehouse"
    val expireOlderThanTimestamp = 1645089917661L

    val conf = new Configuration()
    // 添加DistributedFileSystem类，否则会报异常：org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
    conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem")
    val hadoopCatalog = new HadoopCatalog(conf, icebergWarehousePath)

    val table: Table = hadoopCatalog.loadTable(TableIdentifier.of("iceberg_db", "my_user"))
    table.expireSnapshots().expireOlderThan(expireOlderThanTimestamp).commit()

  }
}

4. 运行程序

将程序进行打包，上传到服务器，然后运行

[root@flink1 ~]# java -classpath flink_dev-1.0-SNAPSHOT.jar flink_test
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[root@flink1 ~]#

5. 查看结果

再次查看最新版本的vN.metadata.json，显示的snapshot-log部分如下：

  "snapshot-log" : [ {
    "timestamp-ms" : 1645089917661,
    "snapshot-id" : 242516093407225541
  }, {
    "timestamp-ms" : 1645089922843,
    "snapshot-id" : 6992061996353419515
  }, {
    "timestamp-ms" : 1645090105742,
    "snapshot-id" : 4375332003125072899
  }, {
    "timestamp-ms" : 1645090219342,
    "snapshot-id" : 4792595715782867813
  } ]

• 可以看到，timestamp-ms小于1645089917661的snapshot-id，不会出现在新的metadata文件中了