#1.CheckpointFunction
Flink 中持久化的动作就是checkpoint ,其在Flink中的所占的分量不言而喻,当我们使用Flink的一些自定义逻辑接口的时候如果在实现逻辑的同时还能实现其 CheckpointFunction接口逻辑,无疑是我们的自定义实现更加趋于完美,同时也有效的体现了Flink 的state 计算的强大能力。
public interface CheckpointedFunction {
/**
* This method is called when a snapshot for a checkpoint is requested. This acts as a hook to the function to
* ensure that all state is exposed by means previously offered through {@link FunctionInitializationContext} when
* the Function was initialized, or offered now by {@link FunctionSnapshotContext} itself.
*
* @param context the context for drawing a snapshot of the operator
* @throws Exception
*/
void snapshotState(FunctionSnapshotContext context) throws Exception;
/**
* This method is called when the parallel function instance is created during distributed
* execution. Functions typically set up their state storing data structures in this method.
*
* @param context the context for initializing the operator
* @throws Exception
*/
void initializeState(FunctionInitializationContext context) throws Exception;
}
#snapshotState方法:当每次任务触发checkpoint时执行,更新保存状态数据
#initializeState方法:初始化checkpoint 存储结构,一般在这里我们会实现两个逻辑:
- 1.判断checkpoint 是否是重启状态恢复,并实现状态恢复逻辑
- 2.初始化checkpoint存储逻辑规则。
FlinkKafkaConsumerBase 源码分析解读
下面将该实现加上响应的注释,以方便大家对这块代码的理解
/** Data for pending but uncommitted offsets. 新的状态快照暂时存储的集合 */
private final LinkedMap pendingOffsetsToCommit = new LinkedMap();
/** Accessor for state in the operator state backend.*/
// 状态存储的规则集合。(简单来说也就是定义了:FlinkKafkaConsumer存储的格式是什么 )
private transient ListState<Tuple2<KafkaTopicPartition, Long>> unionOffsetStates;
/** State name of the consumer's partition offset states. */
private static final String OFFSETS_STATE_NAME = "topic-partition-offset-states";
@Override
public final void initializeState(FunctionInitializationContext context) throws Exception {
//获取计算状态存储对象
OperatorStateStore stateStore = context.getOperatorStateStore();
//通过默认的状态存储名称,获取对应的存储状态集合(如果是初次启动,则一定为空,此操作是为了判断是否为状态恢复操作)
ListState<Tuple2<KafkaTopicPartition, Long>> oldRoundRobinListState =
stateStore.getSerializableListState