1. 异常详情
使用flink 1.11的kafka connector读取0.9版本的kafka报错:
[10:49:12:644] [Source: TableSourceScan(table=[[default_catalog, default_database, test_topic]], fields=[logtime, url, sign, scene, channel]) -> Sink: Sink(table=[default_catalog.default_database.print_table], fields=[logtime, url, sign, scene, channel]) (1/8)] [WARN] - org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleServerDisconnect(NetworkClient.java:1024) - [Consumer clientId=consumer-test_20201101-8, groupId=test_20201101] Bootstrap broker ******* (id: -1 rack: null) disconnected
使用flink 1.7.1的kafka connector读取0.9版本的kafka报错:
10:28:31.098 [Source: Custom Source (8/8)] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-7, groupId=test_20201110] Give up sending metadata request since no node is available
10:28:31.098 [Source: Custom Source (3/8)] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-5, groupId=test_20201110] Connection with /10.202.209.74 disconnected
java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:562)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:498)
at org.apache.kafka.common.network.Selector.poll(Selector.java:427)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
at org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:292)
at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1774)
at org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1742)
at org.apache.flink.streaming.connectors.kafka.internal.KafkaPartitionDiscoverer.getAllPartitionsForTopics(KafkaPartitionDiscoverer.java:77)
at org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.discoverPartitions(AbstractPartitionDiscoverer.java:131)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:473)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
at java.lang.Thread.run(Thread.java:748)
2. 报错原因及解决办法
Flink 1.11版本flink-connector-kafka仅支持kafka 10和11版本,笔者测试使用的kafka为09版本,因此报错。
相关的pom依赖
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka_2.11</artifactId> <version>1.11.0</version> </dependency>
3. Flink与Kafka的版本匹配
下图是flink1.7版本官方文档中关于flink与kafka版本匹配的内容:Apache Kafka Connector
可知,flink-connector-kafka的通用版(即不指定具体kafka版本)支持1.0.0以后的kafka版本。低版本的kafka,如08,09等版本还是需要指定kafka版本的connector,如:flink-connector-kafka-0.9。因此,具体使用的时候要特别注意版本的匹配,否则会发生很多意想不到的错误,导致生产/消费不到数据。