Flink 源码分析之 Flink 是如何 kafka 读取数据的

首先来看一下 FlinkKafkaConsumerBase.run方法，相当于是Flink 从kafka中拉取数据的入口方法:



//入口方法 start a source

public void run(SourceContext<T> sourceContext) throws Exception {

......

// from this point forward:

// - 'snapshotState' will draw offsets from the fetcher,

// instead of being built from `subscribedPartitionsToStartOffsets`

// - 'notifyCheckpointComplete' will start to do work (i.e. commit offsets to

// Kafka through the fetcher, if configured to do so)

//创建Fetcher 从kafka中拉取数据

this.kafkaFetcher = createFetcher(

sourceContext,

subscribedPartitionsToStartOffsets,

periodicWatermarkAssigner,

punctuatedWatermarkAssigner,

(StreamingRuntimeContext) getRuntimeContext(),

offsetCommitMode,

getRuntimeContext().getMetricGroup().addGroup(KAFKA_CONSUMER_METRICS_GROUP),

useMetrics);

if (!running) {

return;

}

// depending on whether we were restored with the current state version (1.3),

// remaining logic branches off into 2 paths:

// 1) New state - partition discovery loop executed as separate thread, with this

// thread running the main fetcher loop

// 2) Old state - partition discovery is disabled and only the main fetcher loop is executed

if (discoveryIntervalMillis == PARTITION_DISCOVERY_DISABLED) {

//未配置KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS

kafkaFetcher.runFetchLoop();

} else {

//仍然调用了kafkaFetcher.runFetchLoop();

runWithPartitionDiscovery();

}

}

createFetcher方法



@Override

protected AbstractFetcher<T, ?> createFetcher(

SourceContext<T> sourceContext,

Map<KafkaTopicPartition, Long> assignedPartitionsWithInitialOffsets,

SerializedValue<AssignerWithPeriodicWatermarks<T>> watermarksPeriodic,

SerializedValue<AssignerWithPunctuatedWatermarks<T>> watermarksPunctuated,

StreamingRuntimeContext runtimeContext,

OffsetCommitMode offsetCommitMode,

Metr