flinkcdc on yarn任务迁移到 on k8s 问题

版本背景

flink:1.14.3

flinkcdc 2.3.0 /2.2.0

现象:

java.lang.NullPointerException: Name is null
	at java.lang.Enum.valueOf(Enum.java:236) ~[?:1.8.0_312]
	at com.ververica.cdc.connectors.mysql.source.offset.BinlogOffsetKind.valueOf(BinlogOffsetKind.java:26) ~[flink-connector-mysql-cdc-2.3.0.1.jar:2.3.0]
	at com.ververica.cdc.connectors.mysql.source.offset.BinlogOffset.getOffsetKind(BinlogOffset.java:136) ~[flink-connector-mysql-cdc-2.3.0.1.jar:2.3.0]
	at com.ververica.cdc.connectors.mysql.source.utils.SerializerUtils.readBinlogPosition(SerializerUtils.java:73) ~[flink-connector-mysql-cdc-2.3.0.1.jar:2.3.0]
	at com.ververica.cdc.connectors.mysql.source.utils.SerializerUtils.readBinlogPosition(SerializerUtils.java:59) ~[flink-connector-mysql-cdc-2.3.0.1.jar:2.3.0]
	at com.ververica.cdc.connectors.mysql.source.split.MySqlSplitSerializer.deserializeSplit(MySqlSplitSerializer.java:153) ~[flink-connector-mysql-cdc-2.3.0.1.jar:2.3.0]
	at com.ververica.cdc.connectors.mysql.source.split.MySqlSplitSerializer.deserialize(MySqlSplitSerializer.java:122) ~[flink-connector-mysql-cdc-2.3.0.1.jar:2.3.0]
	at com.ververica.cdc.connectors.mysql.source.split.MySqlSplitSerializer.deserialize(MySqlSplitSerializer.java:46) ~[flink-connector-mysql-cdc-2.3.0.1.jar:2.3.0]
	at org.apache.flink.core.io.SimpleVersionedSerialization.readVersionAndDeSerialize(SimpleVersionedSerialization.java:165) ~[flink-core-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.util.SimpleVersionedListState$DeserializingIterator.next(SimpleVersionedListState.java:138) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_312]
	at org.apache.flink.util.CollectionUtil.iterableToList(CollectionUtil.java:95) ~[flink-core-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.SourceOperator.open(SourceOperator.java:251) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:585) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:565) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540) ~[flink-streaming-java_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759) ~[flink-runtime_2.11-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-runtime_2.11-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]

原因:

k8s环境cdc版本和yarn环境版本不一致,无法正常从savepoint启动

解决方案:

1.将k8s镜像中版本修改为2.2.0

2.在sink端可以保证一致性的情况写,使用指定时间戳消费mysql数据,减小风险

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值