记录一次阿里云flink生产故障

2022.12.9下午三点半发现flink监控报警一直显示作业反复重启,打开管理平台发现作业在不断自动重启,找到了第一行报错日志.

 

报错日志如下

2022-12-09 15:29:28,263 ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource  [] - Error during binlog processing. Last offset stored = null, binlog reader near position = mysql-bin.001675/185277142
2022-12-09 15:29:28,264 ERROR io.debezium.pipeline.ErrorHandler                            [] - Producer failure
io.debezium.DebeziumException: null
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at java.lang.Thread.run(Thread.java:877) [?:1.8.0_302]
Caused by: java.lang.NullPointerException
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.readMetadata(TableMapEventDataDeserializer.java:80) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:42) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:27) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 3 more
2022-12-09 15:29:28,276 ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource  [] - Error during binlog processing. Last offset stored = null, binlog reader near position = mysql-bin.001675/185277142
2022-12-09 15:29:28,276 ERROR io.debezium.pipeline.ErrorHandler                            [] - Producer failure
io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1670570968000, eventType=UPDATE_ROWS, serverId=1938206218, headerLength=19, dataLength=734, nextPosition=185278024, flags=0}
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) [blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at java.lang.Thread.run(Thread.java:877) [?:1.8.0_302]
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1670570968000, eventType=UPDATE_ROWS, serverId=1938206218, headerLength=19, dataLength=734, nextPosition=185278024, flags=0}
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:309) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:232) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 3 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.MissingTableMapEventException: No TableMapEventData has been found for table id:406820. Usually that means that you have started reading binary log 'within the logical event group' (e.g. from WRITE_ROWS and not proceeding TABLE_MAP
	at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:109) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserializeRows(UpdateRowsEventDataDeserializer.java:71) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:58) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:33) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:232) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 3 more
2022-12-09 15:29:28,665 ERROR org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager [] - Received uncaught exception.
java.lang.RuntimeException: SplitFetcher thread 768 received unexpected exception while polling the records
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:146) ~[flink-table_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
	at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101) [flink-table_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_302]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_302]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
	at java.lang.Thread.run(Thread.java:877) [?:1.8.0_302]
Caused by: org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
	at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.ververica.cdc.connectors.mysql.debezium.task.context.MySqlErrorHandler.setProducerThrowable(MySqlErrorHandler.java:72) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 1 more
Caused by: io.debezium.DebeziumException
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1193) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:958) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 1 more
Caused by: java.lang.NullPointerException
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.readMetadata(TableMapEventDataDeserializer.java:80) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:42) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:27) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:281) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:228) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$1.nextEvent(MySqlStreamingChangeEventSource.java:233) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:945) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850) ~[blob_p-ab098b2976a0009b1eaabcfae48660640b2f63ab-efc656c3c9ffddd2404de700a0f810c2:?]
	... 1 more

TableMapEventDataDeserializer,第80行代码如下

 ColumnType枚举如下

 拉取到mysql-bin.001675/185277142,用阿里云开发提供的测试类读取,打断点进入发现columnTypes[i] & 0xFF的值为20,并不在上述枚举类中,switch(null)会导致报错.

在github上找到了相关issue,原因是mysql8.0.18多了一个类型MYSQL_TYPE_TYPED_ARRAY,这个是不可见的,然后他的ID从244变成了20,导致解析不了,检查了生产的mysql配置,发现确实是8.0.18.

https://github.com/mysql/mysql-server/commit/9082b6a820f3948fd563cc32a050f5e8775f2855

 接下来就是等待阿里云flink发布新版vvr解决问题了

  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值