一、源码提取说明
### --- flink-kafka 是如何消费的?以及如何分区分配等
~~~ open方法源码:
### --- 指定offset提交模式
~~~ OffsetCommitMode:
~~~ OffsetCommitMode:表示偏移量如何从外部提交回Kafka brokers/
~~~ Zookeeper的行为它的确切值是在运行时在使用者子任务中确定的。
~~~ # 源码提取说明:OffsetCommitMode.java
~~~ # 第29~39行
~~~ DISABLED:完全禁用offset提交。
~~~ ON_CHECKPOINTS:只有当检查点完成时,才将偏移量提交回Kafka。
~~~ KAFKA_PERIODIC:使用内部Kafka客户端的自动提交功能,定期将偏移量提交回Kafka。
/**
* The offset commit mode represents the behaviour of how offsets are externally committed
* back to Kafka brokers / Zookeeper.
*
* <p>The exact value of this is determined at runtime in the consumer subtasks.
*/
@Internal
public enum OffsetCommitMode {
/** Completely disable offset committing. */
DISABLED,
/** Commit offsets back to Kafka only when checkpoints are completed. */
ON_CHECKPOINTS,
/** Commit offsets periodically back to Kafka, using the auto commit functionality of internal Kafka clients. */
KAFKA_PERIODIC;
}
~~~ # 源码提取说明:OffsetCommitModes.java
~~~ # 第37~50行
public static OffsetCommitMode fromConfiguration(
boolean enableAutoCommit,
boolean enableCommitOnCheckpoint,
boolean enableCheckpointing) {
if (enableCheckpointing) {
// if checkpointing is enabled, the mode depends only on whether committing on checkpoints is enabled
return (enableCommitOnCheckpoint) ? OffsetCommitMode.ON_CHECKPOINTS : OffsetCommitMode.DISABLED;
} else {
// else, the mode depends only on whether auto committing is enabled in the provided Kafka properties
return (enableAutoCommit) ? OffsetCommitMode.KAFKA_PERIODIC : OffsetCommitMode.DISABLED;
}
}
}
~~~ 使用多个配置值确定偏移量提交模式
~~~ 如果启用了checkpoint,并且启用了checkpoint完成时提交offset,返回ON_CHECKPOINTS。
~~