概述
- Flink1.11 在上周二正式release了,在此之前我也给大家分享过了Flink1.11的一些新特性,然后和大家说过这一期会给大家单独说Flink X Hive
- 本来打算找点数据,然后做一期类似于实时数仓的内容,但是数据不太好找,加上时间、精力有限就简单和大家聊聊吧
- 在开始之前,大家参考一下Hive Integration,把flink 连接hive所需要的包放到lib目录下
Hive Streaming Sink
-
先看看官网是怎么描述Hive Streaming Sink的吧
SET table.sql-dialect=hive; CREATE TABLE hive_table ( user_id STRING, order_amount DOUBLE ) PARTITIONED BY (dt STRING, hr STRING) STORED AS parquet TBLPROPERTIES ( 'partition.time-extractor.timestamp-pattern'='$dt $hr:00:00', 'sink.partition-commit.trigger'='partition-time', 'sink.partition-commit.delay'='1 h', 'sink.partition-commit.policy.kind'='metastore,success-file' ); SET table.sql-dialect=default; CREATE TABLE kafka_table ( user_id STRING, order_amount DOUBLE, log_ts TIMESTAMP(3), WATERMARK FOR log_ts AS log_ts - INTERVAL '5' SECOND ) WITH (...); -- streaming sql, insert into hive table INSERT INTO TABLE hive_table SELECT user_id, order_amount, DATE_FORMAT(log_ts, 'yyyy-MM-dd'), DATE_FORMAT(log_ts, 'HH') FROM kafka_table; -- batch sql, select with partition pruning SELECT * FROM hive_table WHERE dt='2020-05-20' and hr='12';
-
参数给大家稍微解析一下
partition.time-extractor.timestamp-pattern
:分区时间抽取器,与DDL中的分区字段保持一致sink.partition-commit.trigger
:分区触发器类型,需要Source表中定义watermark,当watermark > 提取到的分区时间+sink.partition-commit.delay
中定义的时间,那么就将当前分区提交sink.partition-commit.delay
:相当于延时时间吧sink.partition-commit.policy.kind
:怎么提交,一般提交成功之后,需要通知metastore,这样hive才能读到你最新分区的数据;如果需要合并小文件,也可以自定义Class,通过实现PartitionCommitPolicy接口
-
说了这么多,光说不干假把式,下面就让我们来试一下Hive Streaming Sink
-
先搞个Source,让我们来试一下新的connector