flink源码分析之Broadcast State

最新推荐文章于 2024-09-18 17:11:55 发布

yyqq188

最新推荐文章于 2024-09-18 17:11:55 发布

阅读量292

点赞数

文章标签： java flink

本文链接：https://blog.csdn.net/yyqq188/article/details/113172287

版权

注意事项

broadcast state主要适用于规则匹配这样的场景中,规则的数据量较小且规则的变更频率较少.

而且broadcast state目前的statebackend不支持rockdb这样的存储,而只有本地内存的存储方式,所以要注意内存的适用.

Broadcast State实现的一般步骤

1 首先要有两个DataStream ,一个数据流,一个规则流
DataStream<Action> actions = env.addSource(new KafkaConsumer<>());
DataStream<Pattern> patterns = env.addSource(new KafkaConsumer<>());
2 对数据流进行转化,且对规则转化为广播流
KeyedStream<Action,Long> actionsByUser = actions.keyBy((KeySelector<Action,Long>) action -> action.userId);
MapStateDescriptor<Void,Pattern> bcStateDescriptor = new MapStateDescriptor<>(
"patterns",Types.VOID,Types.POJO(Pattern.class))
BroadcastStream<Pattern> bcPatterns = pattern.broadcast(bcStateDescriptor)
3 对数据流和广播流进行connect操作,并自定义process处理逻辑
DataStream<Tuple2<Long,Pattern>> matches = actionsByUser.connect(bcPatterns)
.process(new PatternEvaluator());
4 PatternEvaluator是具体的处理逻辑,包括三个方法
open:对状态的初始化
processElement:对数据流的处理
processBroadcastElement:对规则流的更新操作

PatternEvaluator具体处理逻辑解析

public static class PatternEvaluator extends ProcessBroadcastFunction<Long,Action,Pattern,Tuple2<Long,Pattern>>{
    //前一刻状态
    ValueState<String> preActionState;
    //广播状态描述符
    MapStateDescriptor<Void,Pattern> patternDesc;

    public void open(Configuration conf){
        //初始化keyedState并注册
        preActionState = getRuntionContext().getState(
            new ValueStateDescriptor("lastAction",Types.STRING)
        )
        patternDesc = new MapStateDescriptor<>("pattern",Types.VOID,Types.POJO(
        Pattern.class));
    }
}
public void processElement(
             Action action,
             ReadOnlyContext ctx,
             Collector<Tuple2<Long,Pattern>> out)throws Exception{
    //从当前的broadcast state中获得pattern
    Pattern pattern = ctx.getBroadcastState(this.patternDesc)
                         .get(null);
    //获得之前的行为
    String preAction = preActionState.value();
    if(patter != null && preAction != null){
        //将pattern和前一行为以及当前行为对比,是否符合规则
        if(pattern.firstAction.equals(preAction) &&
           pattern.secondActon.equals(action.action)){
            out.collect(new Tuple2<>(ctx.getCurrentKey(),pattern));
        }
    }
    //不论匹配与否都要更新当前的action
    preActionState.update(action.action);
    
}
//可变的上下文环境
public void processBroadcastElement(
                            Pattern pattern,
                            Context ctx,
                            Collector<Tuple2<Long,Pattern>> out){
    BroadcastState<Void,Pattern> bcState = ctx.getBroadcastState(patternDesc);
    bcState.put(null,pattern);

}

代码段小部件

[Dàimǎ duàn xiǎo bùjiàn]

Code segment widget