Flink sql 写ddl连接kafka

云端小飞熊

已于 2022-11-08 00:15:08 修改

阅读量3.4k

点赞数 1

分类专栏：代码文章标签： kafka java 大数据

于 2022-11-08 00:13:16 首次发布

本文链接：https://blog.csdn.net/weixin_44777314/article/details/127741927

版权

Flink sql 写ddl连接kafka踩坑日记

背景：flink_2.12-1.14.3
kafka_2.12-3.1.0

准备工作

上传flink的kafka连接器（flink-connector-kafka）
将下载好的 flink-connector-kafka的jar包放在flink的lib目录下启动flink程序
编写flinksql的DDL语句

#kafka的读取ddl
CREATE TABLE source (
	name              STRING,
	age               STRING
) WITH (
	'connector' = 'kafka',
	'topic' = 'test',
	'properties.bootstrap.servers' = 'ip:prot',
	'properties.group.id' = 'group_test',
	'scan.startup.mode' = 'latest-offset',
	'format' = 'json'
);
#kafka的写入ddl
CREATE TABLE sink (
  	name   STRING,
  	sex  STRING
) WITH (
    'connector' = 'kafka',
    'topic' = 'test1',
    'properties.bootstrap.servers' = 'ip:prot',
    'format' = 'json'
);
#业务逻辑实现
insert into sink select * from source;

用bin/sql-client.sh -f xxx.sql 启动脚本提交任务

！！！可能出现的问题：
1.Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/OffsetResetStrategy
这是缺少kafka-clients的jar包，只需将下载的kafka-clients jar包放在lib目录下重启flink，提交任务

Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
	at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
	at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/OffsetResetStrategy
	at org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer.earliest(OffsetsInitializer.java:147)
	at org.apache.flink.connector.kafka.source.KafkaSourceBuilder.<init>(KafkaSourceBuilder.java:104)
	at org.apache.flink.connector.kafka.source.KafkaSource.builder(KafkaSource.java:117)
	at org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSource.createKafkaSource(KafkaDynamicSource.java:379)
	at org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSource.getScanRuntimeProvider(KafkaDynamicSource.java:216)
	at org.apache.flink.table.planner.connectors.DynamicSourceUtils.validateScanSource(DynamicSourceUtils.java:452)
	at org.apache.flink.table.planner.connectors.DynamicSource

最低0.47元/天解锁文章