Flink消费Kafka并统计报警

模拟场景

服务端异常时, 向kafka推送一条ALERT消息, flink解析消息, 如果连续10s钟有3次alert消息, 则发出报警消息

实现方法

在首次获取到alert消息后, 注册一个10s后的timer事件, 后续alert消息, 校验当前时间是否在timer有效期内, 如果不在则重置timer, 待timer到期时, 校验个数是否满足, 如果满足发出报警

代码

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.common.state.ValueState;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.KeyedProcessFunction;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.flink.util.Collector;

import java.util.Properties;

public class ConsumeKafkaAndAlertTest {

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.enableCheckpointing(10 * 1000L);

        Properties prop = new Properties();
        prop.setProperty("bootstrap.servers", "localhost:9092"); // kafka地址
        prop.setProperty("group.id", "consumer_flink_1");

        String consumeTopic = "flink-source"; // topic
        FlinkKafkaConsumer<String> kafkaConsumer = new FlinkKafkaConsumer<>(
                consumeTopic, new SimpleStringSchema(), prop);
        kafkaConsumer.setStartFromLatest();

        env.addSource(kafkaConsumer)
        		// 过滤消息包含ALERT和:的字符串
                .flatMap(new FlatMapFunction<String, String>() {
                    @Override
                    public void flatMap(String s, Collector<String> collector) throws Exception {
                        if (s.contains(" ALERT ") && s.contains(": ")) {
                            collector.collect(s.split(": ")[1]); // 保留 msg
                        }
                    }
                })
                .keyBy(new KeySelector<String, String>() {
                    @Override
                    public String getKey(String s) throws Exception {
                        return "ALERT"; // 使用默认值keyBy
                    }
                }).process(new MyProcessFunction()).print();

        env.execute("My Flink Test");
    }

    private static class MyProcessFunction extends KeyedProcessFunction<String, String, String> {

		// 开始时间
        private ValueState<Long> timer;
        // 从开始到现在一共多少次报警
        private ValueState<Integer> count;
        // 记录各报警消息内容
        private ValueState<String> msg;

        private final int alertStillSeconds = 10;
        private final int alertGreaterThan = 3;

        @Override
        public void open(Configuration parameters) throws Exception {
       		// 初始化消息
            super.open(parameters);
            timer = getRuntimeContext().getState(new ValueStateDescriptor<>("alert_timer", Types.LONG));
            count = getRuntimeContext().getState(new ValueStateDescriptor<>("alert_count", Types.INT));
            msg = getRuntimeContext().getState(new ValueStateDescriptor<>("alert_msg", Types.STRING));
        }

        @Override
        public void processElement(String value, Context ctx, Collector<String> out) throws Exception {
            if (timer == null || timer.value() == null || timer.value() == 0 
                    || timer.value() < ctx.timerService().currentProcessingTime() - alertStillSeconds * 1000) {
                // 添加timer报警 并更新时间
                long pt = ctx.timerService().currentProcessingTime();
                ctx.timerService().registerProcessingTimeTimer(pt + alertStillSeconds * 1000);
                timer.update(pt);
                count.update(1);
                msg.update(value + "\n");
            } else {
                count.update(count.value() + 1);
                msg.update(msg.value() + value + "\n");
            }
        }

        @Override
        public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
            super.onTimer(timestamp, ctx, out);
            // 定时器出发 校验次数是否达到阈值
            if (count.value() >= alertGreaterThan) {
                out.collect("服务器报警.... " + count.value() + "次 msg: \n" + msg.value());
                // 报警完重置
                timer.clear();
                count.clear();
                msg.clear();
            }
        }
    }
}

生产消息

D:\data\kafka_2.12-2.8.0\bin\windows>kafka-console-producer.bat --broker-list 127.0.0.1:9092 --topic flink-source
> ALERT : Hello World
> ALERT : Hello World
> ALERT : Hello World
> ALERT : Hello World
> ALERT : Hello World...
> ALERT : Hello World...
> ALERT : Hello World...

运行结果

Connected to the target VM, address: '127.0.0.1:51010', transport: 'socket'
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
4> 服务器报警.... 4次 msg: 
Hello World
Hello World
Hello World
Hello World

4> 服务器报警.... 3次 msg: 
Hello World...
Hello World...
Hello World...