flink 学习（十）常用的窗口函数

---

前言

记录一下几个简单的窗口函数

1.ReduceFunction

归约

（1）数据源

public class IntegerSource implements SourceFunction<Integer> {
    int i = 0;

    @Override
    public void run(SourceContext ctx) throws Exception {
        while (true) {
            System.out.println(++i);
            ctx.collect(i);
            Thread.sleep(1000);
        }
    }

    @Override
    public void cancel() {

    }
}

（2）示例

 @Test
    public void reduceFunctionTest() throws Exception {
        SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setRuntimeMode(RuntimeExecutionMode.STREAMING)
                .setParallelism(1);

        //添加数据源
        env.addSource(new IntegerSource())
                .windowAll(TumblingProcessingTimeWindows.of(Time.seconds(3)))
                .reduce(new ReduceFunction<Integer>() {
                    @Override
                    public Integer reduce(Integer value1, Integer value2) throws Exception {
                        return value1 + value2;
                    }
                })
                .print("reduce");
        env.execute("ReduceFunction");
    }

结果：

1
2
3
reduce> 6
4
5
6
reduce> 15
7
8
9
reduce> 24

2.AggregateFunction

聚合

示例，求平均值

 @Test
    public void aggregationFunctionTest() throws Exception {
        SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setRuntimeMode(RuntimeExecutionMode.STREAMING)
                .setParallelism(1);

        //添加数据源
        env.addSource(new IntegerSource())
                .windowAll(TumblingProcessingTimeWindows.of(Time.seconds(3)))
                .aggregate(new AggregateFunction<Integer, Tuple2<Integer, Integer>, Integer>() {
                    /**
                     * 累加器
                     */
                    @Override
                    public Tuple2<Integer, Integer> createAccumulator() {
                        return new Tuple2<>(0, 0);
                    }

                    /**
                     * 聚合过程
                     */
                    @Override
                    public Tuple2<Integer, Integer> add(Integer value, Tuple2<Integer, Integer> accumulator) {
                        return new Tuple2<>(value + accumulator.f0, accumulator.f1 + 1);
                    }

                    /**
                     * 处理结果，这里是求平均值
                     */
                    @Override
                    public Integer getResult(Tuple2<Integer, Integer> accumulator) {
                        return accumulator.f0 / accumulator.f1;
                    }

                    /**
                     * 分区合并
                     */
                    @Override
                    public Tuple2<Integer, Integer> merge(Tuple2<Integer, Integer> a, Tuple2<Integer, Integer> b) {
                        return new Tuple2<>(a.f0 + b.f0, a.f1 + b.f1);
                    }

                })
                .print("reduce");
        env.execute("ReduceFunction");
    }

结果：

1
2
3
avg> 1
4
5
6
avg> 4
7
8
avg> 7
9
10
11
avg> 10
12
13
14
avg> 13

3.ProcessWindowFunction

（1）数据源

public class IntegerSource implements SourceFunction<Integer> {

    @Override
    public void run(SourceContext ctx) throws Exception {
        Random random = new Random();
        while (true) {
            int i = random.nextInt(5);
            System.out.println(i);
            ctx.collect(i);
            Thread.sleep(500);
        }
    }

    @Override
    public void cancel() {

    }
}

（2）示例

5秒内每个数字出现的个数

 @Test
    public void processWindowFunctionTest() throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
        DataStreamSource<Integer> source = env.addSource(new IntegerSource());
        //基于ProcessingTime的滚动窗口，窗口长度时3s
        source
                .keyBy(new KeySelector<Integer, Tuple2<Integer, Integer>>() {
                    @Override
                    public Tuple2<Integer, Integer> getKey(Integer value) throws Exception {
                        return new Tuple2<>(value, 1);
                    }
                })
                .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
                .process(new ProcessWindowFunction<>() {
                    @Override
                    public void process(Tuple2<Integer, Integer> integerIntegerTuple2, Context context, Iterable<Integer> elements, Collector<Object> out) throws Exception {
                        System.out.println("key: " + integerIntegerTuple2.f0);
                        Iterator<Integer> it = elements.iterator();
                        int count = 0;
                        while (it.hasNext()) {
                            it.next();
                            count += 1;
                        }
                        out.collect(count);
                    }

                })
                .print("count");
        env.execute("ProcessWindowFunction");
    }

结果：

1
4
2
4
1
0
2
2
4
key: 0
key: 4
key: 1
count:7> 3
count:3> 2
count:6> 1
key: 2
count:3> 3

4.ProcessWindowFunction-ReduceFunction

（1）示例方法

public <R> SingleOutputStreamOperator<R> reduce(
            ReduceFunction<T> reduceFunction, ProcessWindowFunction<T, R, K, W> function) {

通过ReduceFunction进行归约，在通过ProcessWindowFunction处理

（2）数据源

启动nc

nc -lp 8888

（3）示例

分组后通过ReduceFunction计算最大值，再通过ProcessWindowFunction拼接上处理时间

	@Test
    public void ProcessWindowFunctionWithReduceFunctionTest() throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
        DataStreamSource<String> source = env.socketTextStream("172.16.10.159", 8888);
        source
                .map(new MapFunction<String, Tuple2<String, Integer>>() {
                    @Override
                    public Tuple2<String, Integer> map(String value) throws Exception {
                        String[] s = value.split(",");
                        return new Tuple2<>(s[0], Integer.parseInt(s[1]));
                    }
                })
                .keyBy(new KeySelector<Tuple2<String, Integer>, String>() {
                    @Override
                    public String getKey(Tuple2<String, Integer> value) throws Exception {
                        return value.f0;
                    }
                })
                .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
                .reduce(new ReduceFunction<Tuple2<String, Integer>>() {
                            @Override
                            public Tuple2<String, Integer> reduce(Tuple2<String, Integer> value1, Tuple2<String, Integer> value2) throws Exception {
                                return new Tuple2<>(value1.f0, value1.f1 > value2.f1 ? value1.f1 : value2.f1);
                            }
                        },
                        new ProcessWindowFunction<Tuple2<String, Integer>, String, String, TimeWindow>() {
                            @Override
                            public void process(String s, Context context, Iterable<Tuple2<String, Integer>> elements, Collector<String> out) throws Exception {
                                long processingTime = context.currentProcessingTime();
                                Tuple2<String, Integer> tuple2 = elements.iterator().next();
                                out.collect("processingTime : " + processingTime + ", element: " + tuple2);
                            }
                        })
                .print("max");
        env.execute("ProcessWindowFunctionWithReduceFunction");
    }

（3）输入数据

小明,100
小明,90
小月,80
小月,70

（4）结果

max:8> processingTime : 1650379795004, element: (小月,80)
max:2> processingTime : 1650379795004, element: (小明,100)

5.ProcessWindowFunction-AggregateFunction

（1）示例方法

public <ACC, V, R> SingleOutputStreamOperator<R> aggregate(
            AggregateFunction<T, ACC, V> aggFunction,
            ProcessWindowFunction<V, R, K, W> windowFunction) 

通过AggregateFunction进行聚合，再通过ProcessWindowFunction处理

（2）示例

数据分组后，通过AggregateFunction求和，再通过ProcessWindowFunction拼接上时间

@Test
    public void ProcessWindowFunctionWithAggregateFunctionTest() throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
        DataStreamSource<String> source = env.socketTextStream("172.16.10.159", 8888);
        source
                .map(new MapFunction<String, Tuple2<String, Integer>>() {
                    @Override
                    public Tuple2<String, Integer> map(String value) throws Exception {
                        String[] s = value.split(",");
                        return new Tuple2<>(s[0], Integer.parseInt(s[1]));
                    }
                })
                .keyBy(new KeySelector<Tuple2<String, Integer>, String>() {
                    @Override
                    public String getKey(Tuple2<String, Integer> value) throws Exception {
                        return value.f0;
                    }
                })
                .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
                .aggregate(new AggregateFunction<Tuple2<String, Integer>, Tuple2<String, Integer>, Tuple2<String, Integer>>() {
                               @Override
                               public Tuple2<String, Integer> createAccumulator() {
                                   return new Tuple2<>("default", 0);
                               }

                               @Override
                               public Tuple2<String, Integer> add(Tuple2<String, Integer> value, Tuple2<String, Integer> accumulator) {
                                   return new Tuple2<>(value.f0, value.f1 + accumulator.f1);
                               }

                               @Override
                               public Tuple2<String, Integer> getResult(Tuple2<String, Integer> accumulator) {
                                   return accumulator;
                               }

                               @Override
                               public Tuple2<String, Integer> merge(Tuple2<String, Integer> a, Tuple2<String, Integer> b) {
                                   return new Tuple2<>(a.f0, a.f1 + b.f1);
                               }
                           },
                        new ProcessWindowFunction<Tuple2<String, Integer>, String, String, TimeWindow>() {
                            @Override
                            public void process(String s, Context context, Iterable<Tuple2<String, Integer>> elements, Collector<String> out) throws Exception {
                                long processingTime = context.currentProcessingTime();
                                Tuple2<String, Integer> tuple2 = elements.iterator().next();
                                out.collect("processingTime : " + processingTime + ", element: " + tuple2);
                            }
                        })
                .print("sum");
        env.execute("ProcessWindowFunctionWithAggregateFunction");
    }

（3）输入数据

开启nc，跟上一个示例用同样的数据

小明,100
小明,90
小月,80
小月,70

（4）结果：

sum:8> (小月,150)
sum:2> (小明,190)

6.WindowFunction

（1）示例方法

 public <R> SingleOutputStreamOperator<R> apply(WindowFunction<T, R, K, W> function)

通过 WindowFunction 对数据进行处理

（2）示例

@Test
    public void WindowFunctionTest() throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
        DataStreamSource<String> source = env.socketTextStream("172.16.10.159", 8888);
        source
                .map(new MapFunction<String, Tuple2<String, Integer>>() {
                    @Override
                    public Tuple2<String, Integer> map(String value) throws Exception {
                        String[] s = value.split(",");
                        return new Tuple2<>(s[0], Integer.parseInt(s[1]));
                    }
                })
                .keyBy(new KeySelector<Tuple2<String, Integer>, String>() {
                    @Override
                    public String getKey(Tuple2<String, Integer> value) throws Exception {
                        return value.f0;
                    }
                })
                .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
                .apply(new WindowFunction<Tuple2<String, Integer>, Object, String, TimeWindow>() {
                    @Override
                    public void apply(String s, TimeWindow window, Iterable<Tuple2<String, Integer>> input, Collector<Object> out) throws Exception {
                        Iterator<Tuple2<String, Integer>> it = input.iterator();
                        Tuple2<String, Integer> result = null;
                        while (it.hasNext()) {
                            Tuple2<String, Integer> next = it.next();
                            if (result == null) {
                                result = next;
                            } else {
                                result = new Tuple2<>(next.f0, next.f1 > result.f1 ? result.f1 : next.f1);
                            }

                        }
                        out.collect(result);
                    }
                })
                .print("min");
        env.execute("WindowFunction");
    }

（3）输入数据

开启nc，跟上一个示例用同样的数据

小明,100
小明,90
小月,80
小月,70

（4）结果

min:2> (小明,90)
min:8> (小月,70)