flinksql 添加字段从checkpoint重启的问题

39 篇文章 1 订阅

flinksql 在group by添加字段后,从checkpoint重启会报错,

2022-02-07 18:28:33
java.lang.Exception: Exception while creating StreamOperatorStateContext.
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:441)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_563f38a7d955da9b9b5048f28249c3bd_(3/40) from any of the 1 provided restore options.
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
	... 10 more
Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed when trying to restore heap backend
	at org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.restoreState(HeapKeyedStateBackendBuilder.java:177)
	at org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:111)
	at org.apache.flink.runtime.state.hashmap.HashMapStateBackend.createKeyedStateBackend(HashMapStateBackend.java:131)
	at org.apache.flink.runtime.state.hashmap.HashMapStateBackend.createKeyedStateBackend(HashMapStateBackend.java:73)
	at org.apache.flink.runtime.state.StateBackend.createKeyedStateBackend(StateBackend.java:136)
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
	... 12 more
Caused by: org.apache.flink.util.StateMigrationException: The new key serializer (org.apache.flink.table.runtime.typeutils.RowDataSerializer@4ceb3b3d) must be compatible with the previous key serializer (org.apache.flink.table.runtime.typeutils.RowDataSerializer@4ceb3b3d).
	at org.apache.flink.runtime.state.restore.FullSnapshotRestoreOperation.readMetaData(FullSnapshotRestoreOperation.java:210)
	at org.apache.flink.runtime.state.restore.FullSnapshotRestoreOperation.restoreKeyGroupsInStateHandle(FullSnapshotRestoreOperation.java:171)
	at org.apache.flink.runtime.state.restore.FullSnapshotRestoreOperation.access$100(FullSnapshotRestoreOperation.java:113)
	at org.apache.flink.runtime.state.restore.FullSnapshotRestoreOperation$1.next(FullSnapshotRestoreOperation.java:158)
	at org.apache.flink.runtime.state.restore.FullSnapshotRestoreOperation$1.next(FullSnapshotRestoreOperation.java:140)
	at org.apache.flink.runtime.state.heap.HeapSavepointRestoreOperation.restore(HeapSavepointRestoreOperation.java:115)
	at org.apache.flink.runtime.state.heap.HeapSavepointRestoreOperation.restore(HeapSavepointRestoreOperation.java:57)
	at org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.restoreState(HeapKeyedStateBackendBuilder.java:174)
	... 19 more

1、可以通过在一个字段作为json打包字段做冗余,来避免从checkpoint恢复失败的问题.

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
Flink SQL是一个支持流和批两种模式的分布式计算框架,它能够用于各类大数据场景。Flink SQL从1.9版本开始支持基于SQL的批处理功能,最新版本的Flink SQL 1.14.0也都已经逐渐趋于完善。 对于如何从checkpoint中恢复flink-sql任务,实际上与其他flink任务的恢复方式类似。flink-sql在运行过程中,产生了各种状态,如checkpoint状态、状态后端中的状态,元数据等。当一个flink-sql任务意外停止时,重启该任务会需要使用这些状态信息来恢复任务运行的正确状态。 首先,我们需要选定需要的状态后端。Flink提供了不同的状态后端,如memory、filesystem、rocksDB等,在配置文件中选定所需的状态后端,进而启动flink-sql任务。这样flink-sql任务就会产生一系列状态信息,存储在指定的状态后端中。 其次,我们需要设置checkpoint,以保证flink-sql任务在运行过程中产生的状态信息能够被及时保存。Flink提供了不同的checkpoint触发机制,如时间间隔、数据量等,可以根据具体情况选择。 最后,在flink-sql任务出现异常中断时,可以通过使用之前保存的checkpoint状态信息来恢复flink-sql任务,保证任务持续运行。具体可以使用flink提供的命令行工具或者API进行操作。 需要注意的是,在使用flink-sql重启任务时,要确保数据源的指针位于正确的位置,否则将可能导致脏数据的产生,从而影响计算结果的正确性。

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值