[flink1.14.4]Unable to create a source for reading table ‘default_catalog.default_database.new_buyer

已于 2022-04-14 10:48:57 修改
阅读量1.3w 收藏
点赞数 1
分类专栏： flink 大数据 问题集锦 文章标签： 大数据 flink
于 2022-03-11 16:49:35 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/m0_37306331/article/details/123428873
版权
大数据 同时被 3 个专栏收录
9 篇文章 1 订阅
订阅专栏
问题集锦
5 篇文章 0 订阅
订阅专栏
flink
4 篇文章 1 订阅
订阅专栏

升级flink1.14.4报错 

Caused by: org.apache.flink.table.api.ValidationException: Unable to create a source for reading table 'default_catalog.default_database.new_buyer_trade_order2'  

CAUSED BY: 2022-03-11 16:45:04,169 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.class=org.apache.flink.metrics.influxdb.InfluxdbReporter
2022-03-11 16:45:04,169 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.class=org.apache.flink.metrics.influxdb.InfluxdbReporter
2022-03-11 16:45:04,170 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: state.backend.rocksdb.ttl.compaction.filter.enabled=true
2022-03-11 16:45:04,170 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: state.backend.rocksdb.ttl.compaction.filter.enabled=true
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.db=flink
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.db=flink
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.host=192.168.5.57
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.host=192.168.5.57
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: web.timeout=120000
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: web.timeout=120000
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: akka.ask.timeout=120 s
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: akka.ask.timeout=120 s
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: env.java.opts=-verbose:gc -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:ParallelGCThreads=4 -Duser.timezone=Asia/Shanghai
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: env.java.opts=-verbose:gc -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:ParallelGCThreads=4 -Duser.timezone=Asia/Shanghai
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.port=8086
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: metrics.reporter.influxdb.port=8086
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: akka.watch.heartbeat.interval=10 s
2022-03-11 16:45:04,171 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: akka.watch.heartbeat.interval=10 s
2022-03-11 16:45:04,172 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: slotmanager.taskmanager-timeout=600000
2022-03-11 16:45:04,172 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: slotmanager.taskmanager-timeout=600000
2022-03-11 16:45:04,172 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: containerized.heap-cutoff-min=100
2022-03-11 16:45:04,172 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Dynamic Property set: containerized.heap-cutoff-min=100
建表语句解析成功!connector:upsert-kafka
设置kafka时间撮:upsert-kafka
建表语句解析成功!connector:print
hive conf path:/etc/ecm/hive-conf
hive conf path:/etc/ecm/hive-conf
create iceberg catalog success!
start to run sql:CREATE TABLE new_buyer_trade_order2 (
  database VARCHAR,  `table`  VARCHAR,  type  VARCHAR,  ts   BIGINT,  xid   BIGINT,  xoffset  BIGINT,  data  VARCHAR,    `old`   VARCHAR      
) WITH (
'value.format' = 'json',
'key.format' = 'json',
'properties.bootstrap.servers' = '192.168.8.142:9092,192.168.8.141:9092,192.168.8.143:9092',
'connector' = 'upsert-kafka',
'topic' = 'new_buyer_trade_order2')
start to run sql:create table result_print(   database VARCHAR,  `table`  VARCHAR,  type  VARCHAR,  ts   BIGINT,  xid   BIGINT,  xoffset  BIGINT,  data  VARCHAR,    `old`   VARCHAR )with(     'connector' = 'print' )
start to run sql:insert into result_print select      database,  `table`,  type,  ts,  xid,  xoffset,  database,    `old` from new_buyer_trade_order2

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to create a source for reading table 'default_catalog.default_database.new_buyer_trade_order2'.

Table options are:

'connector'='upsert-kafka'
'key.format'='json'
'properties.bootstrap.servers'='192.168.8.142:9092,192.168.8.141:9092,192.168.8.143:9092'
'topic'='new_buyer_trade_order2'
'value.format'='json'
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: org.apache.flink.table.api.ValidationException: Unable to create a source for reading table 'default_catalog.default_database.new_buyer_trade_order2'.

Table options are:

'connector'='upsert-kafka'
'key.format'='json'
'properties.bootstrap.servers'='192.168.8.142:9092,192.168.8.141:9092,192.168.8.143:9092'
'topic'='new_buyer_trade_order2'
'value.format'='json'
	at org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:150)
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:116)
	at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:82)
	at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438)
	at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570)
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:177)
	at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:169)
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1057)
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1026)
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:301)
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlInsert(SqlToOperationConverter.java:639)
	at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:290)
	at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101)
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableEnvironmentImpl.java:834)
	at com.gegejia.flink.FlinkJobBoot.main(FlinkJobBoot.java:95)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
	... 11 more
Caused by: org.apache.flink.table.api.ValidationException: 'upsert-kafka' tables require to define a PRIMARY KEY constraint. The PRIMARY KEY specifies which columns should be read from or write to the Kafka message key. The PRIMARY KEY also defines records in the 'upsert-kafka' table should update or delete on which keys.
	at org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaDynamicTableFactory.validatePKConstraints(UpsertKafkaDynamicTableFactory.java:261)
	at org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaDynamicTableFactory.validateSource(UpsertKafkaDynamicTableFactory.java:219)
	at org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaDynamicTableFactory.createDynamicTableSource(UpsertKafkaDynamicTableFactory.java:119)
	at org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:147)
	... 37 more

 source表未加主键导致，注释放开，提交成功

轩子小姐姐
关注
专栏目录
FlinkSql bug : Column types of query result and sink for registered table ‘default_catalog.******
Direction_Wind的博客
12-07 1052
无
Flink Could not find any factory for identifier ‘json‘ that implements DeserializationFormatFactory
SmartSi
04-09 2129
1. 现象 在运行 Flink SQL 程序时，抛出如下异常： Exception in thread "main" org.apache.flink.table.api.ValidationException: Unable to create a source for reading table 'default_catalog.default_database.kafka_source_table'. Table options are: 'connector'='kafka' 'format'=
Flink 官网table api例子异常排查
百物易用是苏生
07-13 6671
下载的flink 1.13.1版本，scala 2.11版本 按照官网，https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/try-flink/table_api/ 下载https://github.com/apache/flink-playgrounds这个代码 第一个异常： Error: A JNI error has occurred, please check your installation and .
The application contains no execute() calls.
Jhon_yh的博客
12-03 2344
The application contains no execute() calls. 没有调用 execute() 方法。 2021-12-03 09:20:31 [flink-akka.actor.default-dispatcher-4] WARN ApplicationDispatcherBootstrap:169 - Application failed unexpectedly: java.util.concurrent.CompletionException: org.apache.fl
Caused by: org.apache.flink.table.api.ValidationException: Could not find any factory for identifier
weixin_47681855的博客
11-09 3028
报错内容如下： SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" org.apache.
org.apache.spark.SparkException: Unable to create database default as failed to create its directory
gucapg的专栏
03-28 2143
idea + spark在win10系统本地调试中，出现以下错误： Exception in thread "main" org.apache.spark.SparkException: Unable to create database default as failed to create its directory file:/D:/My%2520Works/dev/pro/git/dhe/...
org.apache.flink.table.api.ValidationException: Could not find any factory for identifier ‘doris‘ th
格格颜心
08-01 3162
org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'doris' 打包时需要将多个构件中的class文件或资源文件聚合。资源的聚合需要shade插件提供了丰富的Transformer工具类。
Apache Flink（flink-1.14.4-bin-scala_2.12.tgz）
05-07
Apache Flink（flink-1.14.4-bin-scala_2.12.tgz）是由Apache软件基金会开发的开源流处理框架，其核心是用Java和Scala编写的分布式流数据流引擎。Flink以数据并行和流水线方式执行任意流数据程序，Flink的流水线运行...
Apache Flink（flink-1.14.4-bin-scala_2.11.tgz）
05-07
Apache Flink（flink-1.14.4-bin-scala_2.11.tgz）是由Apache软件基金会开发的开源流处理框架，其核心是用Java和Scala编写的分布式流数据流引擎。Flink以数据并行和流水线方式执行任意流数据程序，Flink的流水线运行...
使用了flink官方示例，尽然提交任务后报错了
热门推荐
github_56967035的博客
06-26 1万+
使用了flink官方示例，尽然提交任务后报错了 背景：参考flink的官方示例，开发FlinkSQL的ETL应用。ETL应用从kafka消费数据，经过数据处理，将处理后的结果数据下沉存储到Clickhouse中。 问题：工程的pom文件中明明依赖了相关连接器,本地运行正常。但打包提交应用到flink时，却报错确少连接器依赖。 错误日志 当pom.xml中依赖的Kafka连接器依赖放置在Clickhouse连接器前面,报错下面日志： The program finished with the follo
Could not execute CREATE CATALOG............
paulgeorge9527的博客
08-09 1986
org.apache.flink.table.api.ValidationException: Could not execute CREATE CATALOG: (catalogName: [hive_catalog], properties: [{hive-conf-dir=/usr/local/apache-hive-3.1.2-bin/conf, default-database=test, type=hive}]) at org.apache.flink.table.api.internal.Ta
flink创建表报错
最新发布
spade_Kwo的博客
01-12 618
例如：随着人工智能的不断发展，机器学习这门技术也越来越重要，很多人都开启了学习机器学习，本文就介绍了机器学习的基础内容。提示：以下是本篇文章正文内容，下面案例可供参考。
flink.table.api.ValidationException: Cannot resolve field [timeConsuming], input field list:[level]
cbb
07-16 1410
flink table api 报错的原因是 在使用table api 进行数据 group by 之后，做数据的排序。这样写之后，运行 报错。 // 数据提取 Table sourceTable = tEnv.from("kafka_source"); Table table = sourceTable.groupBy($("level")) .select($("msg")); 详细错误信息如下： Exception in t
flink-sqlclient报错
科学的N次方
04-19 1534
flink-sqlclient报错： org.apache.flink.table.api.TableException: Table sink ‘default_catalog.default_database.sink’ doesn’t support consuming update changes which is produced by node GroupAggregate 解决方案： 修改sql-client-defaults.yaml中# ‘batch’ or ‘streaming’ exe
Table sink ‘default_catalog.default_database.t3‘ doesn‘t support consuming update and delete changes
m0_37759590的博客
11-10 1560
Table sink 'default_catalog.default_database.t3' doesn't support consuming update and delete changes
Flink SQL JSON Format 源码解析
JasonLee实时计算
10-29 4825
用 Flink SQL 解析 JSON 格式的数据是非常简单的,只需要在 DDL 语句中设置 Format 为 json 即可,像下面这样:CREATETABLEkafka_sourc...
flink sql 从mysql读数据写入到ElasticSearch实践
pop_xiaohao的专栏
12-03 3834
背景 主要是实践下从mysql读数据写入到ElasticSearch,以及总结下相关问题和想法 参考链接：https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/table/connectors/elasticsearch.html 先决条件 添加maven依赖，从mysql读取数据这两个依赖必不可少 <dependency> <groupId>org.apac
flink sql - 问题系列与解决系列
a123147abc的博客
11-30 6926
flink sql 调试-注意点1、布尔类型的坑2、cdc 表[kafka/pg等]，要写对表主键，特别是flink sql 有group by 的情况 1、布尔类型的坑 数据库 deleted = 'false' , flink sql 要替换为 deleted is false 2、cdc 表[kafka/pg等]，要写对表主键，特别是flink sql 有group by 的情况 现象：a 表主键是三个字段的联合主键[PRIMARY KEY (id,parent_id,entity_farm_
failed to execute job 'insert-into_default_catalog.default_database.my_sink
10-23
“failed to execute job 'insert-into_default_catalog.default_database.my_sink”是一个错误消息，通常出现在使用Flink或其他分布式计算框架进行数据处理时。这个错误消息表示作业无法成功执行，并且可能有多个原因导致。 首先，检查作业的代码是否存在错误。可能存在语法错误、逻辑问题或其他错误，导致作业执行失败。查看作业的日志文件，尝试找到错误消息或异常堆栈跟踪，以确定问题所在。 其次，检查作业所需的资源是否足够。可能存在作业需要的内存、CPU或其他资源不足，导致作业无法成功执行。增加作业所需资源的配额或重新分配资源，以确保作业能够正常执行。 还有可能是由于底层数据源或目标出现了问题。检查数据源是否可用，并且提供的连接参数是否正确。同样地，检查目标是否可用，并且接收器的连接参数是否正确。如果有必要，联系相关团队或管理员以解决这些问题。 最后，检查作业的配置是否正确。作业的配置决定了作业如何执行，可能存在配置错误导致作业无法成功执行。检查作业的配置文件或相关配置选项，确保它们符合预期并且没有错误。 综上所述，“failed to execute job 'insert-into_default_catalog.default_database.my_sink”错误可能是由代码错误、资源不足、数据源或目标问题以及配置错误等多种原因导致的。通过仔细检查和排查可能的问题，可以找到并解决这个错误，使作业能够成功执行。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值