Flink 源码分析之 - 如何保存 offset

Flink对Offset的管理，有两种方式：

1.Checkpointing disabled 完全依赖于kafka自身的API

2.Checkpointing enabled 当checkpoint做完的时候，会将offset提交给kafka or zk

本文只针对于第二种，Checkpointing enabled



FlinkKafkaConsumerBase中的 notifyCheckpointComplete

@Override

//当checkpoint完成的时候，此方法会被调用

public final void notifyCheckpointComplete(long checkpointId) throws Exception {

if (!running) {

LOG.debug("notifyCheckpointComplete() called on closed source");

return;

}

final AbstractFetcher<?, ?> fetcher = this.kafkaFetcher;

if (fetcher == null) {

LOG.debug("notifyCheckpointComplete() called on uninitialized source");

return;

}

if (offsetCommitMode == OffsetCommitMode.ON_CHECKPOINTS) {

// only one commit operation must be in progress

if (LOG.isDebugEnabled()) {

LOG.debug("Committing offsets to Kafka/ZooKeeper for checkpoint " + checkpointId);

}

try {

final int posInMap = pendingOffsetsToCommit.indexOf(checkpointId);

if (posInMap == -1) {

LOG.warn("Received confirmation for unk