使用 Flink Hudi 构建流式数据湖

最新推荐文章于 2024-10-03 21:45:41 发布
最新推荐文章于 2024-10-03 21:45:41 发布
阅读量489 收藏 4
点赞数
文章标签： 大数据 spark hadoop java 数据库
原文链接：https://flink-learning.org.cn/
版权

摘要：本文介绍了 Flink Hudi 通过流计算对原有基于 mini-batch 的增量计算模型不断优化演进。用户可以通过 Flink SQL 将 CDC 数据实时写入 Hudi 存储，且在即将发布的 0.9 版本 Hudi 原生支持 CDC format。主要内容为：

  1. 背景

  2. 增量 ETL

  3. 演示

一、背景


近实时

从 2016 年开始，Apache Hudi 社区就开始通过 Hudi 的 UPSERT 能力探索近实时场景的使用案例 [1]。通过 MR/Spark 的批处理模型，用户可以实现小时级别的数据注入 HDFS/OSS。在纯实时场景，用户通过流计算引擎 Flink + KV/OLAP 存储的架构可以实现端到端的秒级 (5分钟级) 实时分析。然而在秒级 (5分钟级) 到小时级时的场景还存在大量的用例，我们称之为 NEAR-REAL-TIME (近实时)。

在实践中有大量的案例都属于近实时的范畴：

  1. 分钟级别的大屏；

  2. 各种 BI 分析 (OLAP)；

  3. 机器学习分钟级别的特征提取。

增量计算

解决近实时的方案当前是比较开放的。

  • 流处理的时延低，但是 SQL 的 pattern 比较固定，查询端的能力（索引、ad hoc）欠缺；

  • 批处理的数仓能力丰富但是数据时延大。

于是 Hudi 社区提出基于 mini-batch 的增量计算模型：

增量数据集 => 增量计算结果 merge 已存结果 => 外存

这套模型通过湖存储的 snapshot 拉取增量的数据集 (两个 commits 之前的数据集)，通过 Spark/Hive 等批处理框架计算增量的结果 (比如简单的 count) 再 merge 到已存结果中。

核心问题

增量模型需要解决的核心问题：

  • UPSERT 能力：类似 KUDU 和 Hive ACID，Hudi 也提供了分钟级的更新能力；

  • 增量消费：Hudi 通过湖存储的多 snapshots 提供增量拉取。

基于 mini-batch 的增量计算模型可以提升部分场景的时延、节省计算成本，但有一个很大的限制：对 SQL 的 pattern 有要求。因为计算走的是批，批计算本身不维护状态，这就要求计算的指标能够比较方便地 merge，简单的 count、sum 可以做，但是 avg、count distinct 这些还是需要拉取全量数据重算。

随着流计算和实时数仓的普及，Hudi 社区也在积极的拥抱变化，通过流计算对原有基于 mini-batch 的增量计算模型不断优化演进：在 0.7 版本引入了流式数据入湖，在 0.9 版本支持了原生的 CDC format。

二、增量 ETL


DB 数据入湖

随着 CDC 技术的成熟，debezium 这样的 CDC 工具越来越流行，Hudi 社区也先后集成了流写，流读的能力。用户可以通过 Flink SQL 将 CDC 数据实时写入 Hudi 存储：

  • 用户既可以通过 Flink CDC connector 直接将 DB 数据导入 Hudi；

  • 也可以先将 CDC 数据导入 Kafka，再通过 Kafka connector 导入 Hudi。

第二种方案的容错和扩展性会好一些。

数据湖 CDC

在即将发布的 0.9 版本，Hudi 原生支持 CDC format，一条 record 的所有变更记录都可以保存，基于此，Hudi 和流计算系统结合的更加完善，可以流式读取 CDC 数据 [2]：

源头 CDC 流的所有消息变更都在入湖之后保存下来，被用于流式消费。Flink 的有状态计算实时累加计算结果 (state)，通过流式写 Hudi 将计算的变更同步到 Hudi 湖存储，之后继续对接 Flink 流式消费 Hudi 存储的 changelog， 实现下一层级的有状态计算。近实时端到端 ETL pipeline：

这套架构将端到端的 ETL 时延缩短到分钟级，并且每一层的存储格式都可以通过 compaction 压缩成列存（Parquet、ORC）以提供 OLAP 分析能力，由于数据湖的开放性，压缩后的格式可以对接各种查询引擎：Flink、Spark、Presto、Hive 等。

一张 Hudi 数据湖表具备两种形态：

  • 表形态：查询最新的快照结果，同时提供高效的列存格式。

  • 流形态：流式消费变更，可以指定任意点位流读之后的 changelog。

三、演示

我们通过一段 Demo 演示 Hudi 表的两种形态。

环境准备

  • Flink SQL Client

  • Hudi master 打包 hudi-flink-bundle jar

  • Flink 1.13.1

这里提前准备一段 debezium-json 格式的 CDC 数据：

{"before":null,"after":{"id":101,"ts":1000,"name":"scooter","description":"Small 2-wheel scooter","weight":3.140000104904175},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606100,"transaction":null}
{"before":null,"after":{"id":102,"ts":2000,"name":"car battery","description":"12V car battery","weight":8.100000381469727},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":null,"after":{"id":103,"ts":3000,"name":"12-pack drill bits","description":"12-pack of drill bits with sizes ranging from #40 to #3","weight":0.800000011920929},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":null,"after":{"id":104,"ts":4000,"name":"hammer","description":"12oz carpenter's hammer","weight":0.75},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":null,"after":{"id":105,"ts":5000,"name":"hammer","description":"14oz carpenter's hammer","weight":0.875},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":null,"after":{"id":106,"ts":6000,"name":"hammer","description":"16oz carpenter's hammer","weight":1},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":null,"after":{"id":107,"ts":7000,"name":"rocks","description":"box of assorted rocks","weight":5.300000190734863},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":null,"after":{"id":108,"ts":8000,"name":"jacket","description":"water resistent black wind breaker","weight":0.10000000149011612},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":null,"after":{"id":109,"ts":9000,"name":"spare tire","description":"24 inch spare tire","weight":22.200000762939453},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":0,"snapshot":"true","db":"inventory","table":"products","server_id":0,"gtid":null,"file":"mysql-bin.000003","pos":154,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1589355606101,"transaction":null}
{"before":{"id":106,"ts":6000,"name":"hammer","description":"16oz carpenter's hammer","weight":1},"after":{"id":106,"ts":10000,"name":"hammer","description":"18oz carpenter hammer","weight":1},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1589361987000,"snapshot":"false","db":"inventory","table":"products","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":362,"row":0,"thread":2,"query":null},"op":"u","ts_ms":1589361987936,"transaction":null}
{"before":{"id":107,"ts":7000,"name":"rocks","description":"box of assorted rocks","weight":5.300000190734863},"after":{"id":107,"ts":11000,"name":"rocks","description":"box of assorted rocks","weight":5.099999904632568},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1589362099000,"snapshot":"false","db":"inventory","table":"products","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":717,"row":0,"thread":2,"query":null},"op":"u","ts_ms":1589362099505,"transaction":null}
{"before":null,"after":{"id":110,"ts":12000,"name":"jacket","description":"water resistent white wind breaker","weight":0.20000000298023224},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1589362210000,"snapshot":"false","db":"inventory","table":"products","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":1068,"row":0,"thread":2,"query":null},"op":"c","ts_ms":1589362210230,"transaction":null}
{"before":null,"after":{"id":111,"ts":13000,"name":"scooter","description":"Big 2-wheel scooter ","weight":5.179999828338623},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1589362243000,"snapshot":"false","db":"inventory","table":"products","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":1394,"row":0,"thread":2,"query":null},"op":"c","ts_ms":1589362243428,"transaction":null}
{"before":{"id":110,"ts":12000,"name":"jacket","description":"water resistent white wind breaker","weight":0.20000000298023224},"after":{"id":110,"ts":14000,"name":"jacket","description":"new water resistent white wind breaker","weight":0.5},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1589362293000,"snapshot":"false","db":"inventory","table":"products","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":1707,"row":0,"thread":2,"query":null},"op":"u","ts_ms":1589362293539,"transaction":null}
{"before":{"id":111,"ts":13000,"name":"scooter","description":"Big 2-wheel scooter ","weight":5.179999828338623},"after":{"id":111,"ts":15000,"name":"scooter","description":"Big 2-wheel scooter ","weight":5.170000076293945},"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1589362330000,"snapshot":"false","db":"inventory","table":"products","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":2090,"row":0,"thread":2,"query":null},"op":"u","ts_ms":1589362330904,"transaction":null}
{"before":{"id":111,"ts":16000,"name":"scooter","description":"Big 2-wheel scooter ","weight":5.170000076293945},"after":null,"source":{"version":"1.1.1.Final","connector":"mysql","name":"dbserver1","ts_ms":1589362344000,"snapshot":"false","db":"inventory","table":"products","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":2443,"row":0,"thread":2,"query":null},"op":"d","ts_ms":1589362344455,"transaction":null}

通过 Flink SQL Client 创建表用来读取 CDC 数据文件：

Flink SQL> CREATE TABLE debezium_source(
>   id INT NOT NULL,
>   ts BIGINT,
>   name STRING,
>   description STRING,
>   weight DOUBLE
> ) WITH (
>   'connector' = 'filesystem',
>   'path' = '/Users/chenyuzhao/workspace/hudi-demo/source.data',
>   'format' = 'debezium-json'
> );
[INFO] Execute statement succeed.

执行 SELECT 观察结果，可以看到一共有 20 条记录，中间有一些 UPDATE s，最后一条消息是 DELETE：

Flink SQL> select * from debezium_source;+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+| op |          id |                   ts |                           name |                    description |                         weight |+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+| +I |         101 |                 1000 |                        scooter |          Small 2-wheel scooter |              3.140000104904175 || +I |         102 |                 2000 |                    car battery |                12V car battery |              8.100000381469727 || +I |         103 |                 3000 |             12-pack drill bits | 12-pack of drill bits with ... |              0.800000011920929 || +I |         104 |                 4000 |                         hammer |        12oz carpenter's hammer |                           0.75 || +I |         105 |                 5000 |                         hammer |        14oz carpenter's hammer |                          0.875 || +I |         106 |                 6000 |                         hammer |        16oz carpenter's hammer |                            1.0 || +I |         107 |                 7000 |                          rocks |          box of assorted rocks |              5.300000190734863 || +I |         108 |                 8000 |                         jacket | water resistent black wind ... |            0.10000000149011612 || +I |         109 |                 9000 |                     spare tire |             24 inch spare tire |             22.200000762939453 || -U |         106 |                 6000 |                         hammer |        16oz carpenter's hammer |                            1.0 || +U |         106 |                10000 |                         hammer |          18oz carpenter hammer |                            1.0 || -U |         107 |                 7000 |                          rocks |          box of assorted rocks |              5.300000190734863 || +U |         107 |                11000 |                          rocks |          box of assorted rocks |              5.099999904632568 || +I |         110 |                12000 |                         jacket | water resistent white wind ... |            0.20000000298023224 || +I |         111 |                13000 |                        scooter |           Big 2-wheel scooter  |              5.179999828338623 || -U |         110 |                12000 |                         jacket | water resistent white wind ... |            0.20000000298023224 || +U |         110 |                14000 |                         jacket | new water resistent white w... |                            0.5 || -U |         111 |                13000 |                        scooter |           Big 2-wheel scooter  |              5.179999828338623 || +U |         111 |                15000 |                        scooter |           Big 2-wheel scooter  |              5.170000076293945 || -D |         111 |                16000 |                        scooter |           Big 2-wheel scooter  |              5.170000076293945 |+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+Received a total of 20 rows

创建  Hudi 表，这里设置表的形态为 MERGE_ON_READ 并且打开 changelog 模式属性 changelog.enabled：

Flink SQL> CREATE TABLE hoodie_table(>   id INT NOT NULL PRIMARY KEY NOT ENFORCED,>   ts BIGINT,>   name STRING,>   description STRING,>   weight DOUBLE> ) WITH (>   'connector' = 'hudi',>   'path' = '/Users/chenyuzhao/workspace/hudi-demo/t1',>   'table.type' = 'MERGE_ON_READ',>   'changelog.enabled' = 'true',>   'compaction.async.enabled' = 'false'> );[INFO] Execute statement succeed.

查询

通过  INSERT 语句将数据导入 Hudi，开启流读模式，并执行查询观察结果：



Flink SQL> select * from hoodie_table/*+ OPTIONS('read.streaming.enabled'='true')*/;+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+| op |          id |                   ts |                           name |                    description |                         weight |+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+| +I |         101 |                 1000 |                        scooter |          Small 2-wheel scooter |              3.140000104904175 || +I |         102 |                 2000 |                    car battery |                12V car battery |              8.100000381469727 || +I |         103 |                 3000 |             12-pack drill bits | 12-pack of drill bits with ... |              0.800000011920929 || +I |         104 |                 4000 |                         hammer |        12oz carpenter's hammer |                           0.75 || +I |         105 |                 5000 |                         hammer |        14oz carpenter's hammer |                          0.875 || +I |         106 |                 6000 |                         hammer |        16oz carpenter's hammer |                            1.0 || +I |         107 |                 7000 |                          rocks |          box of assorted rocks |              5.300000190734863 || +I |         108 |                 8000 |                         jacket | water resistent black wind ... |            0.10000000149011612 || +I |         109 |                 9000 |                     spare tire |             24 inch spare tire |             22.200000762939453 || -U |         106 |                 6000 |                         hammer |        16oz carpenter's hammer |                            1.0 || +U |         106 |                10000 |                         hammer |          18oz carpenter hammer |                            1.0 || -U |         107 |                 7000 |                          rocks |          box of assorted rocks |              5.300000190734863 || +U |         107 |                11000 |                          rocks |          box of assorted rocks |              5.099999904632568 || +I |         110 |                12000 |                         jacket | water resistent white wind ... |            0.20000000298023224 || +I |         111 |                13000 |                        scooter |           Big 2-wheel scooter  |              5.179999828338623 || -U |         110 |                12000 |                         jacket | water resistent white wind ... |            0.20000000298023224 || +U |         110 |                14000 |                         jacket | new water resistent white w... |                            0.5 || -U |         111 |                13000 |                        scooter |           Big 2-wheel scooter  |              5.179999828338623 || +U |         111 |                15000 |                        scooter |           Big 2-wheel scooter  |              5.170000076293945 || -D |         111 |                16000 |                        scooter |           Big 2-wheel scooter  |              5.170000076293945 |

可以看到 Hudi 保留了每行的变更记录，包括 change log 的 operation 类型，这里我们打开 TABLE HINTS 功能，方便动态设置表参数。

继续使用 batch 读模式，执行查询观察输出结果，可以看到中间的变更被合并：

Flink SQL> select * from hoodie_table;2021-08-20 20:51:25,052 INFO  org.apache.hadoop.conf.Configuration.deprecation             [] - mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+| op |          id |                   ts |                           name |                    description |                         weight |+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+| +U |         110 |                14000 |                         jacket | new water resistent white w... |                            0.5 || +I |         101 |                 1000 |                        scooter |          Small 2-wheel scooter |              3.140000104904175 || +I |         102 |                 2000 |                    car battery |                12V car battery |              8.100000381469727 || +I |         103 |                 3000 |             12-pack drill bits | 12-pack of drill bits with ... |              0.800000011920929 || +I |         104 |                 4000 |                         hammer |        12oz carpenter's hammer |                           0.75 || +I |         105 |                 5000 |                         hammer |        14oz carpenter's hammer |                          0.875 || +U |         106 |                10000 |                         hammer |          18oz carpenter hammer |                            1.0 || +U |         107 |                11000 |                          rocks |          box of assorted rocks |              5.099999904632568 || +I |         108 |                 8000 |                         jacket | water resistent black wind ... |            0.10000000149011612 || +I |         109 |                 9000 |                     spare tire |             24 inch spare tire |             22.200000762939453 |+----+-------------+----------------------+--------------------------------+--------------------------------+--------------------------------+Received a total of 10 rows

聚合

Bounded Source 读模式下计算 count(*)：

Flink SQL> select count (*) from hoodie_table;
+----+----------------------+
| op |               EXPR$0 |
+----+----------------------+
| +I |                    1 |
| -U |                    1 |
| +U |                    2 |
| -U |                    2 |
| +U |                    3 |
| -U |                    3 |
| +U |                    4 |
| -U |                    4 |
| +U |                    5 |
| -U |                    5 |
| +U |                    6 |
| -U |                    6 |
| +U |                    7 |
| -U |                    7 |
| +U |                    8 |
| -U |                    8 |
| +U |                    9 |
| -U |                    9 |
| +U |                   10 |
+----+----------------------+
Received a total of 19 rows

Streaming 读模式下计算 count(*)：

Flink SQL> select count (*) from hoodie_table/*+OPTIONS('read.streaming.enabled'='true')*/;
+----+----------------------+
| op |               EXPR$0 |
+----+----------------------+
| +I |                    1 |
| -U |                    1 |
| +U |                    2 |
| -U |                    2 |
| +U |                    3 |
| -U |                    3 |
| +U |                    4 |
| -U |                    4 |
| +U |                    5 |
| -U |                    5 |
| +U |                    6 |
| -U |                    6 |
| +U |                    7 |
| -U |                    7 |
| +U |                    8 |
| -U |                    8 |
| +U |                    9 |
| -U |                    9 |
| +U |                    8 |
| -U |                    8 |
| +U |                    9 |
| -U |                    9 |
| +U |                    8 |
| -U |                    8 |
| +U |                    9 |
| -U |                    9 |
| +U |                   10 |
| -U |                   10 |
| +U |                   11 |
| -U |                   11 |
| +U |                   10 |
| -U |                   10 |
| +U |                   11 |
| -U |                   11 |
| +U |                   10 |
| -U |                   10 |
| +U |                   11 |
| -U |                   11 |
| +U |                   10 |

可以看到 batch 和 streaming 模式下的计算结果是一致的。

浪尖聊大数据-浪尖
关注
新一代流式数据湖平台 Apache Hudi学习笔记 - 基础知识&定义
杨鑫newlife的专栏
04-18 452
什么是Hudi？ 使用Hudi管理数据和基本概念理解，不处理和使用。仅是管理数据。发稿时，当前最新版本是V0.10.0 为什么学习Apache Hudi Hudi是一个数据湖的框架。通过目录和表（分区，列式存储）进行管理。诞生于Uber并开源。 数据仓库 Data Warehouse 简称数仓、DW，是一个用于存储、分析、报告的数据系统。 数据仓库的目的是构建面向分析的集成化数据环境，分析结果是为企业提供决策支持（Decision Support）。 数据仓库的特点是.
使用 Flink Hudi 构建实时数据湖
VqhhMobile的博客
09-19 69
数据湖是一个用于存储和管理各种数据类型的中心化存储系统。它允许组织将结构化和非结构化数据聚合在一起，并支持多种数据处理和分析工作负载。Flink Hudi（Hadoop Upserts Deletes and Incrementals）是一个基于Apache Hudi构建的流式数据湖解决方案，它结合了Apache Flink和Apache Hudi的强大功能，提供了实时数据湖的能力。本文将介绍如何使用Flink Hudi构建实时数据湖，并提供相应的源代码示例。
使用 Flink Hudi 构建流式数据湖平台
Ververica的博客
02-15 1291
▼ 关注「ApacheFlink」，看更多大咖▼摘要：本文整理自阿里巴巴技术专家陈玉兆 (玉兆)、阿里巴巴开发工程师刘大龙 (风离) 在 Flink Forward Asia 2021 ...
《从零起步，开启 Hudi 大数据魔法之旅》
最新发布
m0_74453787的博客
10-03 2085
随着大数据的迅猛发展，企业在数据处理和存储方面面临着越来越多的挑战。Apache Hudi（Hadoop Upserts Deletes and Incrementals）作为一个现代化的大数据框架，旨在解决这些挑战，提供高效的数据湖解决方案。本文将介绍Hudi的基本概念、核心特性以及使用场景。Hudi是一个开源的分布式数据湖框架，主要用于管理和存储大规模数据集。它允许用户在数据湖中进行高效的增量数据更新、删除和查询，同时保持良好的读写性能。
融合趋势下基于 Flink Kylin Hudi 湖仓一体的大数据生态体系
Ververica的博客
04-15 729
摘要：本文由T3 出行大数据平台负责人杨华和资深大数据平台开发工程师王祥虎介绍 Flink、Kylin 和 Hudi 湖仓一体的大数据生态体系以及在 T3的相关应用场景，内容包括：湖仓...
Apache Hudi：新一代流式数据湖平台
SmartSi
08-20 280
早在 2016 年，我们就提出了一个大胆的新愿景 [1]，通过一个新的“增量”数据处理技术栈（结合现有的批处理和流式处理堆栈）重新构想批处理。虽然流处理管道进行面向行的处理，提供秒级处理延迟，但增量管道将对数据湖中的列数据应用相同的原则，高效的数据处理，及相对批处理数量级的改进，同时存储 / 计算可高度扩展。这个新的技术栈将能够毫不费力地支持批量再加工 / 回填的常规处理。Apache Hudi 是作为这一愿景的体现而建立的，它植根于 Uber 面临的真实、困难的问题 [2]，后来在开源社区中独树一帜。总之
大数据Hadoop之——新一代流式数据湖平台 Apache Hudi
匠人精神，持之以恒！
10-16 3999
Hudi（Hadoop Upserts Deletes and Incrementals），简称Hudi，是一个流式数据湖平台，支持对海量数据快速更新，内置表格式，支持事务的存储层、 一系列表服务、数据服务(开箱即用的摄取工具)以及完善的运维监控工具，它可以以极低的延迟将数据快速存储到HDFS或云存储（S3）的工具，最主要的特点支持记录级别的插入更新（Upsert）和删除，同时还支持增量查询。
使用 Flink Hudi 构建流式数据湖平台-Flink Forward Asia 2021.pdf
04-29
通过以上介绍，我们了解到使用 Apache Flink 和 Apache Hudi 构建流式数据湖平台是大数据领域的一种先进实践，它能有效应对大规模、快速变化的数据处理需求，并为数据湖提供强大的数据更新和管理能力。Flink Hudi ...
流式数据湖平台实战 | 在FlinkSQL中集成和使用Hudi
shangjg3的博客
10-07 473
2.插入数据本文介绍在Flink 中集成和使用Hudi。介绍Flink如何将Streaming引入Hudi。在Hudi上使用Flink，并学习Flink读写Hudi的不同模式：Flink SQL客户端写入：Flink SQL客户端写入（读取）Hudi。配置：对于全局配置，通过$FLINK_HOME/conf/FLINK-conf.yaml进行设置。对于每个作业配置，通过表选项进行设置。
基于 Flink SQL 和 Paimon 构建流式湖仓新方案
击水三千里的专栏
12-22 1882
Paimon 结合了湖存储和 LSM（Log Structured MergeTree）技术，提供了低延时、低成本的流式数据湖解决方案。 Paimon 与 Flink 有深度集成，支持 CDC
技术解读｜流式计算实现秒级数据入湖
巨杉数据库技术社区
08-24 405
数据入湖的时效性直接影响整体数据应用效果，基于湖仓一体架构的巨杉数据库通过对接业界主流的Flink,Spark和Storm等主流的流式框架，实现实时生产数据的高速入湖，原汁原味的将数据保留在巨杉数据库中。
如何打造一个流式数据湖
weixin_43857576的博客
12-28 723
利用Flink cdc+Flink+Hudi构建流式数据胡
Flink+hudi 构架沧湖一体化解决方案
微信搜：import_bigdata，大数据领域硬核原创作者
12-10 1798
简介Apache Hudi(发音为"Hoodie")在DFS的数据集上提供以下流原语插入更新 (如何改变数据集？)增量拉取 (如何获取变更的数据？)Hudi维护在数据集上执行的所有操作的时...
ByteLake：字节跳动基于Apache Hudi的实时数据湖平台
Focus on Lakehouse
11-02 1198
一篇关于字节跳动基于 Apache Hudi 的实时数据湖平台 ByteLake 的分享。本篇内容包含四个部分，首先介绍一下 Hudi，其次介绍字节的实时数据湖平台 ByteLake 的应...
基于数据湖格式构建流式增量数仓—CDC
阿里云云栖号
02-22 821
今天以 Apache Hudi 为主，分享一下数据湖格式上实现 CDC 的一些思考和注意点，以及一些流式 streaming 通用的优化点，和 Hudi CDC 的后续规划。
Flink SQL与Paimon联手：构建高效流式湖仓新架构
例如，Flink可以与Apache Iceberg或Apache Hudi等数据湖存储系统协同工作，这些系统支持诸如Append格式（如FormatV2）、MergeInto操作和高效的更新策略（如Upsert和PositionDeleteFile/EqualityDeleteFile）。...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值