1、概述
当指定了大于 0 的 allowed lateness 时,窗口本身以及其中的内容仍会在 watermark 越过窗口末端后保留;
如果一个迟到但未被丢弃的数据到达,它可能会再次触发这个窗口,这种触发被称作 late firing;
late firing 发出的元素应该被视作对之前计算结果的更新,即你的数据流中会包含一个相同计算任务的多个结果,应用需要考虑到这些重复的结果,或去除重复的部分。
2、代码示例
package com.xu.flink.datastream.day09;
import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.windowing.WindowFunction;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;
import java.time.Duration;
/**
* 当指定了大于 0 的 allowed lateness 时,窗口本身以及其中的内容仍会在 watermark 越过窗口末端后保留;
* 如果一个迟到但未被丢弃的数据到达,它可能会再次触发这个窗口,这种触发被称作 late firing。
*
* late firing 发出的元素应该被视作对之前计算结果的更新,即你的数据流中会包含一个相同计算任务的多个结果,应用需要考虑到这些重复的结果,或去除重复的部分。
*/
public class _15_WindowGetLatenessData {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource<String> input = env.socketTextStream("localhost", 8888);
// 测试时限制了分区数,生产中需要设置空闲数据源
env.setParallelism(2);
OutputTag<Tuple2<String, Long>> lateOutputTag = new OutputTag<Tuple2<String, Long>>("late-data"){};
// 事件时间需要设置水位线策略和时间戳
SingleOutputStreamOperator<Tuple2<String, Long>> map = input.map(new MapFunction<String, Tuple2<String, Long>>() {
@Override
public Tuple2<String, Long> map(String input) throws Exception {
String[] fields = input.split(",");
return new Tuple2<>(fields[0], Long.parseLong(fields[1]));
}
});
SingleOutputStreamOperator<Tuple2<String, Long>> watermarks = map.assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple2<String, Long>>forBoundedOutOfOrderness(Duration.ofSeconds(0))
.withTimestampAssigner(new SerializableTimestampAssigner<Tuple2<String, Long>>() {
@Override
public long extractTimestamp(Tuple2<String, Long> input, long l) {
return input.f1;
}
}));
SingleOutputStreamOperator<String> resStream = watermarks.keyBy(e -> e.f0)
.window(TumblingEventTimeWindows.of(Duration.ofSeconds(5)))
.trigger(EventTimeTrigger.create())
.allowedLateness(Duration.ofSeconds(3))
.sideOutputLateData(lateOutputTag)
.apply(new WindowFunction<Tuple2<String, Long>, String, String, TimeWindow>() {
@Override
public void apply(String s, TimeWindow timeWindow, Iterable<Tuple2<String, Long>> iterable, Collector<String> collector) throws Exception {
System.out.println(timeWindow.getStart() + "-" + timeWindow.getEnd());
for (Tuple2<String, Long> tuple : iterable) {
collector.collect(tuple.f0);
}
}
});
resStream.print("window=>");
resStream.getSideOutput(lateOutputTag).print("late=>");
env.execute();
}
}
3、数据测试
完全迟到的数据会通过侧流输出
a,1718157600000
b,1718157600000
c,1718157600000
a,1718157602000
b,1718157602000
c,1718157602000
a,1718157605001
b,1718157605001
1718157600000-1718157605000
window=>:2> a
window=>:2> a
1718157600000-1718157605000
window=>:1> b
window=>:1> b
1718157600000-1718157605000
window=>:1> c
window=>:1> c
c,1718157605001
a,1718157604000
b,1718157604000
c,1718157604000
1718157600000-1718157605000
window=>:2> a
window=>:2> a
window=>:2> a
1718157600000-1718157605000
window=>:1> b
window=>:1> b
window=>:1> b
1718157600000-1718157605000
window=>:1> c
window=>:1> c
window=>:1> c
a,1718157608001
b,1718157608001
c,1718157608001
a,1718157604001
b,1718157604001
c,1718157604001
late=>:2> (a,1718157604001)
late=>:1> (b,1718157604001)
late=>:1> (c,1718157604001)