问题描述:在对 mysql 某个数据库所有表进行初始化的时候(同步表的历史数据),部分表可以正常初始化,部份表出现循环重复初始化,bootstrap 进入死循环,导致产生大量重复的 bootstrap-insert 消息分发到 kafka。
现象:初始化耗时过长。笔者操作的某库总所有表的记录数在 19w 条左右,初始化进行半个小时还没有结束。进一步确认初始化是否进入死循环,可以查询 maxwell 的 scheam 数据库中的 bootstrap 表,该表记录了各个表已经完成初始化的记录行数:
select database_name,table_name,is_complete,inserted_rows from bootstrap;
重复执行 sql 查询,如果发现某个 table_name 的 inserted_row 值往返出现 0~n、0~n,哦豁,maxwell-bootstrap 已经进入死循环。
查看 Maxwell 运行日志 /usr/local/maxwell-1.26.2/bin/../logs/MaxwellDaemon.out 发现如下异常:
java.sql.SQLException: HOUR_OF_DAY: 2 -> 3
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:85)
at com.mysql.cj.jdbc.result.ResultSetImpl.getTimestamp(ResultSetImpl.java:903)
at com.mysql.cj.jdbc.result.ResultSetImpl.getObject(ResultSetImpl.java:1243)
at com.zendesk.maxwell.bootstrap.SynchronousBootstrapper.setRowValues(SynchronousBootstrapper.java:210)
at com.zendesk.maxwell.bootstrap.SynchronousBootstrapper.performBootstrap(SynchronousBootstrapper.java:69)
at com.zendesk.maxwell.bootstrap.SynchronousBootstrapper.startBootstrap(SynchronousBootstrapper.java:39)
at com.zendesk.maxwell.bootstrap.BootstrapController.work(BootstrapController.java:65)
at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:34)
at com.zendesk.maxwell.MaxwellContext.lambda$getBootstrapController$0(MaxwellContext.java:395)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.cj.exceptions.WrongArgumentException: HOUR_OF_DAY: 2 -> 3
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.mysql.cj.result.SqlTimestampValueFactory.localCreateFromTimestamp(SqlTimestampValueFactory.java:112)
at com.mysql.cj.result.SqlTimestampValueFactory.localCreateFromTimestamp(SqlTimestampValueFactory.java:50)
at com.mysql.cj.result.AbstractDateTimeValueFactory.createFromTimestamp(AbstractDateTimeValueFactory.java:87)
at com.mysql.cj.protocol.a.MysqlTextValueDecoder.decodeTimestamp(MysqlTextValueDecoder.java:79)
at com.mysql.cj.protocol.result.AbstractResultsetRow.decodeAndCreateReturnValue(AbstractResultsetRow.java:87)
at com.mysql.cj.protocol.result.AbstractResultsetRow.getValueFromBytes(AbstractResultsetRow.java:241)
at com.mysql.cj.protocol.a.result.TextBufferRow.getValue(TextBufferRow.java:132)
... 9 more
Caused by: java.lang.IllegalArgumentException: HOUR_OF_DAY: 2 -> 3
at java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2829)
at java.util.Calendar.updateTime(Calendar.java:3393)
at java.util.Calendar.getTimeInMillis(Calendar.java:1782)
at com.mysql.cj.result.SqlTimestampValueFactory.localCreateFromTimestamp(SqlTimestampValueFactory.java:108)
... 15 more
解决办法:修改源代码
克隆源码到本地:链接见文末
- 修改类 MaxwellBootstrapUtilityConfig 的 36 行为:
public String getConnectionURI( ) {
return "jdbc:mysql://" + mysql.host + ":" + mysql.port + "/" + schemaDatabaseName + "?serverTimezone=Asia/Shanghai";
}
2. 修改 pom.xml 文件 mysql 驱动类版本为:
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
</dependency>
然后对项目进行打包并重新部署打包后的 maxwell 到环境:
mvn package
重新初始化,大概 10 分钟即可完成 bootstrap … 奶思~
以上操作基于 maxwell v1.26.2
相关连接:
Maxwell's Daemon
https://github.com/zendesk/maxwell