46、Flink窗口迟到数据的处理 AllowedLateness 代码示例

1、概述

使用 event-time 窗口时，数据可能会迟到，默认 watermark 一旦越过窗口结束的 timestamp，迟到的数据就会被直接丢弃；

Allowed lateness 默认是 0，在 watermark 超过窗口末端、到达窗口末端加上 allowed lateness 之前的这段时间内到达的元素，依旧会被加入窗口；

触发的结果取决于触发器是否 purge 而导致结果不同；

ProcessTime 和 GlobalWindows 无迟到数据。

2、代码示例

import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.windowing.WindowFunction;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.triggers.EventTimeTrigger;
import org.apache.flink.streaming.api.windowing.triggers.PurgingTrigger;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.util.Collector;

import java.time.Duration;

/**
 * 使用 event-time 窗口时，数据可能会迟到，默认 watermark 一旦越过窗口结束的 timestamp，迟到的数据就会被直接丢弃；
 * Allowed lateness 默认是 0，在 watermark 超过窗口末端、到达窗口末端加上 allowed lateness 之前的这段时间内到达的元素，依旧会被加入窗口；
 * 触发的结果取决于触发器是否 purge 而导致结果不同
 *
 * ProcessTime 和 GlobalWindows 无迟到数据
 */
public class _14_WindowAllowedLateness {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        DataStreamSource<String> input = env.socketTextStream("localhost", 8888);

        // 测试时限制了分区数，生产中需要设置空闲数据源
        env.setParallelism(2);

        // 事件时间需要设置水位线策略和时间戳
        SingleOutputStreamOperator<Tuple2<String, Long>> map = input.map(new MapFunction<String, Tuple2<String, Long>>() {
            @Override
            public Tuple2<String, Long> map(String input) throws Exception {
                String[] fields = input.split(",");
                return new Tuple2<>(fields[0], Long.parseLong(fields[1]));
            }
        });

        SingleOutputStreamOperator<Tuple2<String, Long>> watermarks = map.assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple2<String, Long>>forBoundedOutOfOrderness(Duration.ofSeconds(0))
                .withTimestampAssigner(new SerializableTimestampAssigner<Tuple2<String, Long>>() {
                    @Override
                    public long extractTimestamp(Tuple2<String, Long> input, long l) {
                        return input.f1;
                    }
                }));

        watermarks.keyBy(e -> e.f0)
                .window(TumblingEventTimeWindows.of(Duration.ofSeconds(5)))
                .trigger(PurgingTrigger.of(EventTimeTrigger.create()))
//                .trigger(EventTimeTrigger.create())
                .allowedLateness(Duration.ofSeconds(3))
                .apply(new WindowFunction<Tuple2<String, Long>, String, String, TimeWindow>() {
                    @Override
                    public void apply(String s, TimeWindow timeWindow, Iterable<Tuple2<String, Long>> iterable, Collector<String> collector) throws Exception {
                        System.out.println(timeWindow.getStart() + "-" + timeWindow.getEnd());

                        for (Tuple2<String, Long> tuple : iterable) {
                            collector.collect(tuple.f0);
                        }
                    }
                })
                .print();

        env.execute();
    }
}

3、数据测试

1）EventTimeTrigger-不清除窗口内的数据：设置允许延迟的时间后，当迟到的数据到达时，窗口会再次触发

          a,1718157600000
          b,1718157600000
          c,1718157600000
         
          a,1718157602000
          b,1718157602000
          c,1718157602000
         
          a,1718157605001
          b,1718157605001
         
          1718157600000-1718157605000
          2> a
          2> a
          1718157600000-1718157605000
          1> b
          1> b
          1718157600000-1718157605000
          1> c
          1> c
         
          c,1718157605001
         
          a,1718157604000
          b,1718157604000
          c,1718157604000
         
          1718157600000-1718157605000
          2> a
          2> a
          2> a
          1718157600000-1718157605000
          1> b
          1> b
          1> b
          1718157600000-1718157605000
          1> c
          1> c
          1> c

2）PurgingTrigger.of(EventTimeTrigger.create())：清除窗口内的数据，设置允许延迟的时间后，当迟到的数据到达时，窗口会再次触发

          a,1718157600000
          b,1718157600000
          c,1718157600000
         
          a,1718157602000
          b,1718157602000
          c,1718157602000
         
          a,1718157605001
          b,1718157605001
         
          1718157600000-1718157605000
          2> a
          2> a
          1718157600000-1718157605000
          1> b
          1> b
          1718157600000-1718157605000
          1> c
          1> c
         
          c,1718157605001
         
          a,1718157604000
          b,1718157604000
          c,1718157604000
         
          1718157600000-1718157605000
          2> a
          1718157600000-1718157605000
          1> b
          1718157600000-1718157605000
          1> c