感谢您抽出
![d8ecf8f7cdd8d9b7e0826c850ad06ad8.gif](https://i-blog.csdnimg.cn/blog_migrate/5969a66dfbaa0e0f7bce131af2b33a28.gif)
.
![535eeda53630e8537f22b0d62955c5fa.gif](https://i-blog.csdnimg.cn/blog_migrate/504069ef79da0efb91e0969a53c93515.gif)
.
![0e3916616ab817aa003fb7fd1f121978.gif](https://i-blog.csdnimg.cn/blog_migrate/3ae15f8b52d0b9ef17edb58145fd7d73.gif)
来阅读本文
FlinkSql消费Kafka的多分区topic,并实时计算指标,同样的Sql第一次执行和第二次执行结果不一样?
问题描述:
初始条件:Flink版本 1.11,kafka版本0.10,kafka的topic为message-json,3个分区,kafka内没有数据。
在Flink的sql-client端执行创建kafkasource table:
create table iservVisit ( type string comment '时间类型', uuid string comment '用户uri', clientTime string comment '10位时间戳', rowtime as to_timestamp(from_unixtime(cast(substring(coalesce(clientTime, '0'), 1, 10) as bigint))), -- 计算列, 10位时间戳转为timestamp类型 WATERMARK for rowtime as rowtime - INTERVAL '1' MINUTE -- 计算列, 作为watermark) with ( 'connector' = 'kafka', 'topic' = 'message-json', 'properties.bootstrap.servers' = 'localhost:9092', 'properties.group.id' = 'consumer-rt', 'format' = 'json', 'json.ignore-parse-errors' = 'true', 'scan.startup.mode' = 'earliest-offset')
然后再执行如下语句,实时生成指标,通过窗口做pv/uv的计算:
select tumble_start(rowtime, interval '2' MINUTE) as wStart, tumble_end(rowtime, interval '2' MINUTE) as wEnd, count(1) as pv, count(distinct uuid) as uv from iservVisitgroup by tumble(rowtime, interval '2' MINUTE)
这时开始手动一条一条往kafka中发送数据:
kafka记录 clientTime消息时间 产生的watermark时间 说明{"type":