Canal异常[errorNumber=1094, fieldCount=-1...]

最新推荐文章于 2023-05-29 11:30:00 发布
最新推荐文章于 2023-05-29 11:30:00 发布
阅读量1.7k 收藏
点赞数
分类专栏： Canal 数据同步 mysql 文章标签： java Canal ETL
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u010953816/article/details/118723577
版权
mysql 同时被 3 个专栏收录
4 篇文章 0 订阅
订阅专栏
Canal
1 篇文章 0 订阅
订阅专栏
数据同步
1 篇文章 0 订阅
订阅专栏
在项目中遇到了Canal从MySQL获取binlog时出现的异常，具体表现为Canal解析rowdata失败，错误信息提示limit exceed。分析原因是由于MySQL系统表权限过高导致Canal无法读取相关表的binlog，从而无法正常移动位点。解决方案是在Canal配置中增加黑名单，过滤掉mysql库下的所有表，无需重启Canal即可恢复。参考链接提供了解决方案的详细步骤。
摘要由CSDN通过智能技术生成

项目中，遇到的Canal异常如下：

2021-07-14 09:57:30.501 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - connect MysqlConnecti
on to /localhost:3306...
2021-07-14 09:57:30.549 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - handshake initializat
ion packet received, prepare the client authentication packet to send
2021-07-14 09:57:30.549 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - client authentication
 packet is sent out.
2021-07-14 09:57:30.597 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - auth switch response 
packet is sent out.
2021-07-14 09:57:31.076 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.a.otter.canal.parse.inbound.mysql.MysqlConnection - Register slave RegisterSl
aveCommandPacket[reportHost=192.168.1.80,reportPort=40858,reportUser=case,reportPasswd=password,serverId=1560281089,command=21]
2021-07-14 09:57:31.124 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.a.otter.canal.parse.inbound.mysql.MysqlConnection - COM_BINLOG_DUMP with posi
tion:BinlogDumpCommandPacket[binlogPosition=4,slaveServerId=1560281089,binlogFileName=binlog.000022,command=18]
2021-07-14 09:57:31.196 [pool-4-thread-2] INFO  c.a.otter.canal.server.embedded.CanalServerWithEmbedded - getWithoutAck successfully, clientId:1001 batchSize:50  real size is 15 and result is
 [batchId:1 , position:PositionRange[start=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,posit
ion=227,serverId=1,gtid=,timestamp=1626066389000]],ack=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.
000022,position=12561,serverId=1,gtid=,timestamp=1626066755000]],end=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journ
alName=binlog.000022,position=12561,serverId=1,gtid=,timestamp=1626066755000]],endSeq=14]]
2021-07-14 09:57:31.197 [pool-4-thread-2] INFO  c.a.otter.canal.server.embedded.CanalServerWithEmbedded - ack successfully, clientId:1001 batchId:1 position:PositionRange[start=LogPosition[id
entity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,position=227,serverId=1,gtid=,timestamp=1626066389000]],ack=Lo
gPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,position=12561,serverId=1,gtid=,timestamp=16260667
55000]],end=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,position=12561,serverId=1,gtid=,time
stamp=1626066755000]],endSeq=14]
2021-07-14 09:57:31.298 [pool-4-thread-2] INFO  c.a.otter.canal.server.embedded.CanalServerWithEmbedded - getWithoutAck successfully, clientId:1001 batchSize:50  real size is 81 and result is
 [batchId:2 , position:PositionRange[start=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,posit
ion=13685,serverId=1,gtid=,timestamp=1626066777000]],ack=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlo
g.000022,position=70920,serverId=1,gtid=,timestamp=1626068949000]],end=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,jou
rnalName=binlog.000022,position=70920,serverId=1,gtid=,timestamp=1626068949000]],endSeq=95]]
2021-07-14 09:57:31.325 [destination = canal-staging-test , address = /localhost:3306 , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - dump address /120.25.22
5.226:3306 has an error, retrying. caused by 
com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed.
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed.
Caused by: java.lang.IllegalArgumentException: limit excceed: 369
	at com.taobao.tddl.dbsync.binlog.LogBuffer.getUint8(LogBuffer.java:235)
	at com.taobao.tddl.dbsync.binlog.LogBuffer.getPackedLong(LogBuffer.java:1024)
	at com.taobao.tddl.dbsync.binlog.JsonDiffConversion.print_json_diff(JsonDiffConversion.java:48)
	at com.taobao.tddl.dbsync.binlog.event.RowsLogBuffer.fetchValue(RowsLogBuffer.java:1073)
	at com.taobao.tddl.dbsync.binlog.event.RowsLogBuffer.nextValue(RowsLogBuffer.java:125)
	at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseOneRow(LogEventConvert.java:726)
	at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.LogEventConvert.parseRowsEvent(LogEventConvert.java:549)
	at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$DmlParserStage.onEvent(MysqlMultiStageCoprocessor.java:330)
	at com.alibaba.otter.canal.parse.inbound.mysql.MysqlMultiStageCoprocessor$DmlParserStage.onEvent(MysqlMultiStageCoprocessor.java:316)
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-07-14 09:57:31.326 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - disConnect MysqlConne
ction to /localhost:3306...
2021-07-14 09:57:31.326 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - disConnect MysqlConne
ction to /localhost:3306...
2021-07-14 09:57:31.365 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - connect MysqlConnecti
on to /localhost:3306...
2021-07-14 09:57:31.404 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - handshake initializat
ion packet received, prepare the client authentication packet to send
2021-07-14 09:57:31.404 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - client authentication
 packet is sent out.
2021-07-14 09:57:31.440 [pool-4-thread-2] INFO  c.a.otter.canal.server.embedded.CanalServerWithEmbedded - ack successfully, clientId:1001 batchId:2 position:PositionRange[start=LogPosition[id
entity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,position=13685,serverId=1,gtid=,timestamp=1626066777000]],ack=
LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,position=70920,serverId=1,gtid=,timestamp=162606
8949000]],end=LogPosition[identity=LogIdentity[sourceAddress=/localhost:3306,slaveId=-1],postion=EntryPosition[included=false,journalName=binlog.000022,position=70920,serverId=1,gtid=,ti
mestamp=1626068949000]],endSeq=95]
2021-07-14 09:57:31.443 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - auth switch response 
packet is sent out.
2021-07-14 09:57:31.521 [destination = canal-staging-test , address = /localhost:3306 , EventParser] INFO  c.alibaba.otter.canal.parse.driver.mysql.MysqlConnector - KILL DUMP 358092 fail
ure
java.io.IOException: ErrorPacket [errorNumber=1094, fieldCount=-1, message=Unknown thread id: 358092, sqlState=HY000, sqlStateMarker=#]
 with command: KILL CONNECTION 358092
	at com.alibaba.otter.canal.parse.driver.mysql.MysqlUpdateExecutor.update(MysqlUpdateExecutor.java:49)
	at com.alibaba.otter.canal.parse.driver.mysql.MysqlConnector.disconnect(MysqlConnector.java:113)
	at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.disconnect(MysqlConnection.java:97)
	at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:315)
	at java.lang.Thread.run(Thread.java:748)

分析：mysql系统表权限较高，canal读该表的binlog失败，位点无法移动

解决：将配置项中黑名单加上mysql下的所有表：canal.instance.filter.black.regex = mysql\\..* ，修改后canal集群不需要重启即可恢复；

参考：https://www.cnblogs.com/softidea/p/12894008.html

Think_Coding
关注
专栏目录
MYSQL服务器错误代码和消息
枫叶飘零的BLOG
05-06 1914
MYSQL服务器错误代码和消息服务器错误信息来自下述源文件：·         错误消息信息列在share/errmsg.txt文件中。“%d”和“%s”分别代表编号和字符串，显示时，它们将被消息值取代。·         错误值列在share/errmsg.txt文件中，用于生成include/mysqld_error.h和include/mysqld
canal连接RDS MySQL binlog报错收集及解决办法
weixin_42392293的博客
04-21 2373
如果canal获取不到binlog的数据，可以去查看canal目录下的 logs/example/example.log 日志数据。 在项目中我遇到以下错误，记录并分享。希望对你有帮助~ 错误一： Caused by: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW comm...
java.io.IOException: ErrorPacket [errorNumber=1193,||Unknown system variable 'binlog_c……Canal启动报错
-
01-23 3458
自己想要尝试Redis同步MySQL来提升自己的web项目的查找速度，再看了几种连接模式后觉得由阿里使用的Canal查找binlog模式更加稳定符合未来的发展，所以进行了尝试但是其中真的坑很多很多。 一、首先看自己的MySQL的版本如果是5.62以下的话对不起请先升级版本吧 因为：binlog_checksum是MySQL 5.6.2才引入的，5.6.2以下根本就没有这个参数 所以最后一步启动C...
sql执行报错---＞1094 unknow thread id
sias_Amazing的博客
10-16 5910
全表查询数据查询失败（大概有180万条） 报错信息 sql执行报错—>1094 什么unknow thread id 优化并解决 具体原因是一次查询超时，导致sql未执行完并且报错 优化sql 分页查询并且一次只查询1000进行处理，本身就是定时任务，所以耗时并不影响 图片: ...
Canal IOException Error
Asaue的博客
08-24 197
mysql配置canal时一个IOException Error 在根据阿里github上的配置，配置canal的时候出现了一个报错，通过filebeat收集日志分析 如下报错时，更改conf/canal.properties 放开 canal.instance.parser.parallelThreadSize = 16 即可 问题解决，canal的example.log已经没有ERROR输出了。 ...
mysql error 1094,ERROR 1094 (HY000) at line 1: Unknown thread id
weixin_42349005的博客
03-17 2520
/usr/local/bin/mysql -h xxx -u xxx -Pxxx -pxxx -e 'show processlist'|grep -i "select" | awk '{if ($6 > 10) {printf "/usr/local/bin/mysql -h xxx -u xxx -Pxxx -pxxx -e \"kill %d\"\n",$1}}'|sh为什么会提示...
mysql error 1094_同步过程中老是出现kill connection 386810688 error ERROR 1094 (HY000): Unknown thread id: 386...
weixin_42364640的博客
02-06 1702
Variable_name | Value |+---------------------------------------+----------+| connect_timeout | 100 || delayed_insert_timeout | 300 ...
ERROR c.a.otter.canal.parse.inbound.mysql.MysqlConnection - java.io.IOException: ErrorPacket [errorNumber=1193, fieldCount=-1, message=Unknown system variable 'binlog_checksum', sqlState=HY000, sqlStateMarker=#] with command: select @@global.binlog_checksum此报错是什么原因
最新发布
05-31
这个错误是由于 MySQL 数据库版本过低导致的。`binlog_checksum` 是 MySQL 5.6.9 版本中引入的一个系统变量，如果你的 MySQL 版本低于这个版本，就会出现这个错误。 解决方法是升级 MySQL 数据库版本，或者在 MySQL...
canal安装
weixin_40875934的博客
12-07 1730
Mysql 安装 1、安装前，我们可以检测系统是否自带安装 MySQL: rpm -qa | grep mysql ·如果安装的有mysql rpm -e mysql OR rpm -e --nodeps mysql 2、wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm 3、rpm -ivh mysql-co...
十一（1）、MySQL遇到的问题
Angular_need的博客
05-18 409
1、从dump文件中导入表结构报错： ERROR 1840 (HY000) at line 24: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty. https://www.cnblogs.com/tonnytangy/p/7779164.html ...
canal 源码解析（2）-数据流转篇（1）
skycanf的博客
06-19 1039
1） 上一篇 只是正常启动，但是线程是等待中，没有数据接入处理。现在开始模拟同步数据，并分析其中原理  如上一片线程阻塞点位于：AbstractEventParser类的start方法2）mysql主从复制重点就在这里了，当前因为 startposition里部位null，// 4. 开始dump数据 // 判断所属instance是否启用GTID模式，是的话调用ErosaConnection中G...
MYSQL常见出错代码解析
quxiuer
07-15 4922
   1005：MYSQL创建表失败　　　　1006：MYSQL创建数据库失败　　　　1007：MYSQL数据库已存在，创建数据库失败　　　　1008：MYSQL数据库不存在，删除数据库失败　　　　1009：MYSQL不能删除数据库文件导致删除数据库失败　　　　1010：MYSQL不能删除数据目录导致删除数据库失败　　      1011：MYSQL
解决Java中的IOException异常
热门推荐
weixin_45934981的博客
05-29 3万+
IOException是Java中的一个受检查异常（Checked Exception）。它是java.io包中定义的异常类之一，用于表示输入输出操作期间可能发生的错误或异常情况。
Otter：connect /192.168.10.225:3306 failure:java.io.IOException: Error When doing Client
似水流年
10-13 1万+
pid:1 nid:1 exception:canal:canal1:com.alibaba.otter.canal.parse.exception.CanalParseException: java.io.IOException: connect /192.168.10.225:3306 failure:java.io.IOException: Error When doing Client A
canal报错show master status‘ has an error!
zhaoguowei的博客
01-16 3325
记一次dbus-canal线上日志报错解决 dbus表检查数据线，发现dbus数据线不通，数据线显示进程停止 到canal去找日志，解压最后一个日志文件 gzip -d 1.log.gz 2021-01-10 23:58:32.964 [destination = evoicedc , address = /dbus-mysql:3306 , EventParser] ERROR com.alibaba.ot ter.canal.common.alarm.LogAlarmHandler - destinat
java socket 网络闪断_Node节点网络闪断后报异常
weixin_39625563的博客
02-26 542
如果节点的网络短暂(如10秒 - 1分钟)与源数据库的机器连不上，就会重试3次连接网络，然后Kill掉连接。接着网络连接上了，但报错：Could not find first log file name in binary log index file，报3次这样的错后，又Kill掉连接，如此kill掉3次后，最后不停地报：NullPointerException。最后只用停止Channel，删除...
初试canal
jianyuqiuqiu的博客
07-25 846
canal小测试-
java kill dump_canal KILL DUMP
weixin_33941707的博客
02-27 512
position infocanal.instance.master.address = xx.xx.xx.xx:3306canal.instance.master.journal.name =canal.instance.master.position =canal.instance.master.timestamp = 14710764850002016-08-13 16:23:09.821 ...
canal连接mysql失败_Canal启动成功，但是查看instance发现错误
weixin_32573931的博客
03-03 2331
canal用的版本是 v1.1.4mysql 和 canal全部都是用docker启动， 是用docker-compose 管理mysql my.cnf[mysqld]server_id=1log-bin=mysql-bin # 开启 binlogbinlog-format=ROW # 选择 ROW 模式[mysql]default-character-set=utf8instance.p...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值