【Flink】 Flink与Kafka版本对应关系

1.概述

转载：Flink与Kafka版本对应关系

转载这个主要是上次做flink kafka版本升级，忘记记录了。

先上干货：以下为Flink和Kafka的版本对照表

Flink版本	Kafka版本
1.12.X	2.4.1
1.11.X	2.4.1
1.10.X	2.2.1
1.9.X	2.2.0
1.8.X	2.0.1
1.7.X	2.0.1
0.10.x	0.8.2.0
0.9.x	0.8.2.0

最近在使用Flink进行Kafka吞吐量测试时，遇到了以下问题导致程序无法运行
去网上查资料也没有很直接的解决方法，大致也就是说是Flink与Kafka的版本问题
目前Flink采用的是1.12.0版本，Kafka采用的是2.3.0版本

报错信息如下：

Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
	at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:638)
	at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1509)
	at lenrnflink.Te2.main(Te2.java:24)
Caused by: java.lang.NoSuchMethodError: org.apache.flink.api.common.serialization.DeserializationSchema.open(Lorg/apache/flink/api/common/serialization/DeserializationSchema$InitializationContext;)V
	at org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializationSchemaWrapper.open(KafkaDeserializationSchemaWrapper.java:46)
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:696)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Thread.java:748)

pom.xml中关于Flink与Flink-Kafka连接器配置如下：

Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
	at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:638)
	at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1509)
	at lenrnflink.Te2.main(Te2.java:24)
Caused by: java.lang.NoSuchMethodError: org.apache.flink.api.common.serialization.DeserializationSchema.open(Lorg/apache/flink/api/common/serialization/DeserializationSchema$InitializationContext;)V
	at org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializationSchemaWrapper.open(KafkaDeserializationSchemaWrapper.java:46)
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:696)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Thread.java:748)

网上说有可能是Kafka服务器与客户端Fetch版本不一致导致的，然后我经过检查发现服务器与客户端版本Fetch一致。

Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
	at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:638)
	at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1509)
	at lenrnflink.Te2.main(Te2.java:24)
Caused by: java.lang.NoSuchMethodError: org.apache.flink.api.common.serialization.DeserializationSchema.open(Lorg/apache/flink/api/common/serialization/DeserializationSchema$InitializationContext;)V
	at org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializationSchemaWrapper.open(KafkaDeserializationSchemaWrapper.java:46)
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:696)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
	at java.lang.Thread.run(Thread.java:748)

	//客户端fetch版本，0-11，与服务器一致
    public static Schema[] schemaVersions() {
        return new Schema[]{FETCH_REQUEST_V0, FETCH_REQUEST_V1, FETCH_REQUEST_V2, FETCH_REQUEST_V3, FETCH_REQUEST_V4, FETCH_REQUEST_V5, FETCH_REQUEST_V6, FETCH_REQUEST_V7, FETCH_REQUEST_V8, FETCH_REQUEST_V9, FETCH_REQUEST_V10, FETCH_REQUEST_V11};
    }
    

想要去找Flink与Kafka的版本对应关系，发现并没有找到，偶然在Maven发现了线索
在Maven仓库网站https://mvnrepository.com/中，找到flink-connector-kafka的详情页面：https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka

这里的依赖包与Flink版本一一对应，我们点击Version为1.12.0的链接进去，发现页面中有一列为：Compile Dependencies，里面是kafka-client的2.4.1的依赖

说明Flink1.12.0推荐的Kafka版本为2.4.1，但是我使用的Kafka为2.3.0，版本不一致，导致报错。
之后我把Kafka也升级到2.4.1，错误解决
这边建议各位在使用Flink中Kafka连接器的时候，去Maven仓库中查看一下依赖，选用匹配的Flink与Kafka版本，避免出现异常。

最终我们的是Flink主版本是 flink 1.12.4 但是我们的connect 是kafka-connect 1.11.0 这样改动量是最小的，也是支持kafka 0.10.0版本的。

M.扩展

flink引出的kafka不同版本的兼容性