1、概述
使用 event-time 窗口时,数据可能会迟到,默认 watermark 一旦越过窗口结束的 timestamp,迟到的数据就会被直接丢弃;
Allowed lateness 默认是 0,在 watermark 超过窗口末端、到达窗口末端加上 allowed lateness 之前的这段时间内到达的元素,依旧会被加入窗口;
触发的结果取决于触发器是否 purge 而导致结果不同;
ProcessTime 和 GlobalWindows 无迟到数据。
2、代码示例
import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.windowing.WindowFunction;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger;
import org.apache.flink.streaming.api.windowing.triggers.PurgingTrigger;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.util.Collector;
import java.time.Duration;
/**
* 使用 event-time 窗口时,数据可能会迟到,默认 watermark 一旦越过窗口结束的 timestamp,迟到的数据就会被直接丢弃;
* Allowed lateness 默认是 0,在 watermark 超过窗口末端、到达窗口末端加上 allowed lateness 之前的这段时间内到达的元素,依旧会被加入窗口;
* 触发的结果取决于触发器是否 purge 而导致结果不同
*
* ProcessTime 和 GlobalWindows 无迟到数据
*/
public class _14_WindowAllowedLateness {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource<String> input = env.socketTextStream("localhost", 8888);
// 测试时限制了分区数,生产中需要设置空闲数据源
env.setParallelism(2);
// 事件时间需要设置水位线策略和时间戳
SingleOutputStreamOperator<Tuple2<String, Long>> map = input.map(new MapFunction<String, Tuple2<String, Long>>() {
@Override
public Tuple2<String, Long> map(String input) throws Exception {
String[] fields = input.split(",");
return new Tuple2<>(fields[0], Long.parseLong(fields[1]));
}
});
SingleOutputStreamOperator<Tuple2<String, Long>> watermarks = map.assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple2<String, Long>>forBoundedOutOfOrderness(Duration.ofSeconds(0))
.withTimestampAssigner(new SerializableTimestampAssigner<Tuple2<String, Long>>() {
@Override
public long extractTimestamp(Tuple2<String, Long> input, long l) {
return input.f1;
}
}));
watermarks.keyBy(e -> e.f0)
.window(TumblingEventTimeWindows.of(Duration.ofSeconds(5)))
.trigger(PurgingTrigger.of(EventTimeTrigger.create()))
// .trigger(EventTimeTrigger.create())
.allowedLateness(Duration.ofSeconds(3))
.apply(new WindowFunction<Tuple2<String, Long>, String, String, TimeWindow>() {
@Override
public void apply(String s, TimeWindow timeWindow, Iterable<Tuple2<String, Long>> iterable, Collector<String> collector) throws Exception {
System.out.println(timeWindow.getStart() + "-" + timeWindow.getEnd());
for (Tuple2<String, Long> tuple : iterable) {
collector.collect(tuple.f0);
}
}
})
.print();
env.execute();
}
}
3、数据测试
1)EventTimeTrigger-不清除窗口内的数据:设置允许延迟的时间后,当迟到的数据到达时,窗口会再次触发
a,1718157600000
b,1718157600000
c,1718157600000
a,1718157602000
b,1718157602000
c,1718157602000
a,1718157605001
b,1718157605001
1718157600000-1718157605000
2> a
2> a
1718157600000-1718157605000
1> b
1> b
1718157600000-1718157605000
1> c
1> c
c,1718157605001
a,1718157604000
b,1718157604000
c,1718157604000
1718157600000-1718157605000
2> a
2> a
2> a
1718157600000-1718157605000
1> b
1> b
1> b
1718157600000-1718157605000
1> c
1> c
1> c
2)PurgingTrigger.of(EventTimeTrigger.create()):清除窗口内的数据,设置允许延迟的时间后,当迟到的数据到达时,窗口会再次触发
a,1718157600000
b,1718157600000
c,1718157600000
a,1718157602000
b,1718157602000
c,1718157602000
a,1718157605001
b,1718157605001
1718157600000-1718157605000
2> a
2> a
1718157600000-1718157605000
1> b
1> b
1718157600000-1718157605000
1> c
1> c
c,1718157605001
a,1718157604000
b,1718157604000
c,1718157604000
1718157600000-1718157605000
2> a
1718157600000-1718157605000
1> b
1718157600000-1718157605000
1> c