Flink
RayBreslin
大数据开发、设计企业应用
展开
-
Flink sql:Table sink doesn‘t support consuming update and delete changes which is produced by node
一、问题描述Flink sql将kafka作为join的输出,报错:Exception in thread "main" org.apache.flink.table.api.TableException: Table sink 'default_catalog.default_database.ods_t' doesn't support consuming update and delete changes which is produced by node Join(joinType=[LeftO原创 2022-05-05 18:08:37 · 4004 阅读 · 0 评论 -
Flink报错:org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold一、问题描述flink执行任务报错2022-01-11 15:10:49org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold. at org.apache.flink.runtime.c原创 2022-01-14 16:28:07 · 4775 阅读 · 0 评论 -
Flink 侧输出报错:Could not determine TypeInformation for the OutputTag type.
一、问题描述flink侧输出报错 The program finished with the following exception The program finished with the following exception:Could not determine TypeInformation for the OutputTag type. The most common reas原创 2021-11-09 20:07:15 · 2310 阅读 · 0 评论 -
flink集成hudi执行sql报错:ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
一、问题描述flink sql创建hudi表后,执行sql插入数据,结果报错没有hadoop相关包Flink SQL> insert into order_info_hudi select * from order_info;[INFO] Submitting SQL update statement to the cluster...[ERROR] Could not execute SQL statement. Reason:java.lang.ClassNotFoundExceptio原创 2021-10-25 12:53:12 · 3755 阅读 · 6 评论 -
flink报错:Caused by: java.lang.NumberFormatException: Not a version: 9
一、问题描述flink执行CEP的时候,报错Caused by: java.lang.NumberFormatException: Not a version: 9 at scala.util.PropertiesTrait$class.parts$1(Properties.scala:184) at scala.util.PropertiesTrait$class.isJavaAtLeast(Properties.scala:187) at scala.util.Properties$.isJa原创 2021-09-09 19:42:51 · 617 阅读 · 0 评论 -
flink报错:IllegalStateException: No ExecutorFactory found to execute the application
Flink 报错: No ExecutorFactory found to execute the application一、问题描述执行 Flink 1.12.0 程序报错:SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.原创 2021-08-29 13:20:06 · 5261 阅读 · 0 评论 -
flink 1.12.0 提示keyBy警告 Symbol keyBy is deprecated. use [[DataStream.keyBy(KeySelector)]] instead
flink 1.12.0 提示keyBy警告 Symbol keyBy is deprecated. use [[DataStream.keyBy(KeySelector)]] instead一、问题描述flink 1.12相对于1.10个别接口有更新,所以,需要依据代码进行微调。例如keyBy方法[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HyrtAct5-1627635607595)(tmp.assets/1627635349912.png)]二、问题原因ke原创 2021-07-30 17:00:31 · 1346 阅读 · 0 评论 -
flink sql on yarn报错: org.codehaus.janino.CompilerFactory cannot be cast to org.codehaus.commons.comp
一、问题描述使用flink on yarn方式提交Flink sql,抛出如下异常org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to instantiate java compiler at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.j原创 2021-07-29 13:21:49 · 1567 阅读 · 0 评论 -
flink sql报错:Bang equal ‘!=‘ is not allowed under the current SQL conformance level
一、问题描述flink sql当有过滤条件时,写的sql如下, tableEnv.executeSql( s""" |insert into kafka | select user_id from test_ods where user_id!='uid11' |""".stripMargin)报错Caused by: org.apache.calcite.runtime.CalciteException: Bang equa原创 2021-07-29 10:43:21 · 3786 阅读 · 0 评论 -
flink报错:No implicits found for parameter evidence$...
一、问题描述flink程序报错No implicits found for parameter evidence$二、问题原因没有加隐式转换 ,implicit。flink需要手动添加。三、解决办法引入隐式转换import org.apache.flink.api.scala._/原创 2021-06-26 17:55:35 · 1057 阅读 · 4 评论 -
Flink(3):DataSet的Source、Transform和Sink算子,以及计数器
一、概述Flink批处理DataSet的处理流程Source、Transform和Sink算子。二、source【参考:https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/batch/#data-sources】1.基于文件创建(1)readTextFile(path) / TextInputForma...原创 2019-12-04 22:21:56 · 1051 阅读 · 0 评论 -
Flink error:No data sinks have been created yet. A program needs at least one sink that consumes dat
一、问题描述Flink程序,批处理程序,执行报错:Exception in thread "main" java.lang.RuntimeException: No data sinks have been created yet. A program needs at least one sink that consumes data. Examples are writing the...原创 2019-12-04 17:48:57 · 2790 阅读 · 0 评论 -
Flink error:File or directory data/output already exists. Existing files and directories are not ...
一、问题描述flink在sink到本地目录的时候,报错:09:12:19,166 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job DataSet Sink (5e5507e2029038a82605e0ffd0ff4a14) switched from state FAILING to F...原创 2019-12-04 09:25:12 · 2178 阅读 · 0 评论 -
idea:maven无法获取flink函数的源码
一、问题描述在获取flink读取函数readTextFile的源码的时候,一直无法获取。二、问题原因Maven的镜像是默认的,可能因为下载速度或者没有源码,导致无法获取。所以,添加上阿里云的镜像源即可下载。三、 解决办法1.maven的setting.xml中添加阿里云镜像 <mirrors> <mirror> <...原创 2019-12-03 08:40:35 · 550 阅读 · 0 评论 -
Flink(2):实现流处理实例(Java和scala)
一、实现功能使用scala和java分别实现Flink的流式处理。二、实现功能代码1.Java(1)代码package com.bd.flink._1130WordCount;import org.apache.flink.api.common.functions.FlatMapFunction;import org.apache.flink.api.java.tupl...原创 2019-11-30 20:05:52 · 2134 阅读 · 1 评论 -
Flink(1):开发环境的搭建(Java和scala)
一、实现功能流式开发Flink开发环境的搭建。二、实现步骤:Java开发环境【参考官网:https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/projectsetup/java_api_quickstart.html】1.本地环境(1)官网要求Maven 3.0.4 (or higher) and ...原创 2019-11-30 16:37:47 · 1900 阅读 · 0 评论