一、源码提取说明
### --- 直接启动consumer
~~~ # 源码提取说明:FlinkKafkaConsumerBase.java
~~~ # 第160~161行
/** The startup mode for the consumer (default is {@link StartupMode#GROUP_OFFSETS}). */
private StartupMode startupMode = StartupMode.GROUP_OFFSETS;
### --- 该枚举类型有5个值:
~~~ GROUP_OFFSETS:从保存在zookeeper或者是Kafka broker的对应消费者组提交的offset开始消费,
~~~ 这个是默认的配置
~~~ EARLIEST:尽可能从最早的offset开始消费
~~~ LATEST:从最近的offset开始消费
~~~ TIMESTAMP:从用户提供的timestamp处开始消费
~~~ SPECIFIC_OFFSETS:从用户提供的offset处开始消费
### --- 根据startup mode,获取从哪个地方开始消费。然后,partition discoverer就会拉取初始分区的数据
~~~ # 源码提取说明:FlinkKafkaConsumerBase.java
~~~ # 第583~605行
LOG.info("Consumer subtask {} will start reading {} partitions with offsets in restored state: {}",
getRuntimeContext().getIndexOfThisSubtask(), subscribedPartitionsToStartOffsets.size(), subscribedPartitionsToStartOffsets);
} else {
// use the partition discoverer to fetch the initial seed partitions,
// and set their initial offsets depending on the startup mode.
// for SPECIFIC_OFFSETS and TIMESTAMP modes, we set the specific offsets now;
// for other modes (EARLIEST, LATEST, and GROUP_OFFSETS), the offset is lazily determined
// when the partition is actually read.
switch (startupMode) {
case SPECIFIC_OFFSETS:
if (specificStartupOffsets == null) {
throw new IllegalStateException(
"Startup mode for the consumer set to " + StartupMode.SPECIFIC_OFFSETS +
", but no specific offsets were specified.");
}
for (KafkaTopicPartition seedPartition : allPartitions) {
Long specificOffset = specificStartupOffsets.get(seedPartition);
if (specificOffset != null) {
// since the specified offsets represent the next record to read, we subtract
// it by one so that the initial state of the consumer will be correct
subscribedPartitionsToStartOffsets.put(seedPartition, specificOffset - 1);
} else {
~~~ 如果startup模式为SPECIFIC_OFFSETS:
~~~ 异常情况:如果没有配置具体从哪个offset开始消费
### --- 正常情况:获取每个分区指定的消费起始offset
Long specificOffset = specificStartupOffsets.get(seedPartition);
~~~ # 源码提取说明:FlinkKafkaConsumerBa