超级会员免费看
本文介绍了Flink如何实现与Hive的流式读写，包括使用Hive Streaming进行写入，实现实时分区表监控增量读取，并探讨了Hive表作为时态表进行流数据join的方法。Flink支持通过Hadoop OutputFormat/RecordWriter进行流式写入Hive，而读取则分为分区表和非分区表的增量读取策略，但不支持水印语法，可能不适合窗口操作。
0 Hive Streaming
A typical hive job is scheduled periodically to execute, so there will be a large delay.
Flink supports to write, read and join the hive table in the form of streaming.
1 Streaming Writing
The Hive table supports streaming writes, based on Filesystem Streaming Sink.
The Hive Streaming Sink re-use Filesystem Streaming Sink to integrate Hadoop OutputFormat/RecordWriter to streaming writing. Hadoop RecordWriters are Bulk-encoded Formats, Bulk Formats rolls files on every checkpoint.
By default, now only have renaming committer, this means S3 filesystem can not supports exactly-once, if you want to use Hive streaming sink in S3 filesystem, You can configure the following parameter to false to us
订阅专栏 解锁全文
2531
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
