环境
- Seata
0.8.1
问题描述
项目中使用Seata解决分布式事务问题,一开始都很正常,可在一次事务更新的条数增多后,开始出现异常,StackTrace如下:
Caused by: java.lang.RuntimeException: org.springframework.orm.jpa.JpaSystemException: Unable to commit against JDBC Connection; nested exception is org.hibernate.TransactionException: Unable to commit against JDBC Connection
org.springframework.orm.jpa.JpaSystemException: Unable to commit against JDBC Connection; nested exception is org.hibernate.TransactionException: Unable to commit against JDBC Connection
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:351)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:253)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:536)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:746)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:714)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:534)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:305)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.sanfenqiu.innerbuy.dal.agent.prov.impl.order.OrderRepositoryImpl$$EnhancerBySpringCGLIB$$be944e1a.saveForOpen(<generated>)
at com.alibaba.dubbo.common.bytecode.Wrapper88.invokeMethod(Wrapper88.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:76)
at com.alibaba.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:52)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at io.seata.integration.dubbo.alibaba.TransactionPropagationFilter.invoke(TransactionPropagationFilter.java:61)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:62)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:73)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:104)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:96)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:173)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.TransactionException: Unable to commit against JDBC Connection
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:87)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:272)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:104)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:532)
... 39 more
Caused by: java.sql.SQLException: io.seata.core.exception.TransactionException: Response[TransactionException[null]]
at io.seata.rm.datasource.ConnectionProxy.recognizeLockKeyConflictException(ConnectionProxy.java:135)
at io.seata.rm.datasource.ConnectionProxy.processGlobalTransactionCommit(ConnectionProxy.java:184)
at io.seata.rm.datasource.ConnectionProxy.commit(ConnectionProxy.java:161)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.commit(AbstractLogicalConnectionImplementor.java:81)
... 42 more
Caused by: io.seata.core.exception.TransactionException: Response[TransactionException[null]]
at io.seata.rm.AbstractResourceManager.branchRegister(AbstractResourceManager.java:68)
at io.seata.rm.DefaultResourceManager.branchRegister(DefaultResourceManager.java:96)
at io.seata.rm.datasource.ConnectionProxy.register(ConnectionProxy.java:207)
at io.seata.rm.datasource.ConnectionProxy.processGlobalTransactionCommit(ConnectionProxy.java:182)
... 44 more
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:106)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:73)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:138)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:104)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:96)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:173)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 common frames omitted
问题排查
从异常分析得知,问题出在seata身上,且必然出现更新的条数少时没有问题,多时开始报错。定位Seata源码(AbstractResourceManager.java:68)
,发现是RM从TC返回错误,但一个TransactionException[null]
报得不明不白。
@Override
public Long branchRegister(BranchType branchType, String resourceId, String clientId, String xid, String applicationData, String lockKeys) throws TransactionException {
try {
BranchRegisterRequest request = new BranchRegisterRequest();
request.setXid(xid);
request.setLockKey(lockKeys);
request.setResourceId(resourceId);
request.setBranchType(branchType);
request.setApplicationData(applicationData);
BranchRegisterResponse response = (BranchRegisterResponse) RmRpcClient.getInstance().sendMsgWithResponse(request);
if (response.getResultCode() == ResultCode.Failed) {
>>>> throw new TransactionException(response.getTransactionExceptionCode(), "Response[" + response.getMsg() + "]");
}
return response.getBranchId();
} catch (TimeoutException toe) {
throw new TransactionException(TransactionExceptionCode.IO, "RPC Timeout", toe);
} catch (RuntimeException rex) {
throw new TransactionException(TransactionExceptionCode.BranchRegisterFailed, "Runtime", rex);
}
}
开始尝试升级Seata,将seata-all
由原来的v0.8.1
升级为0.9.0.1
,然而问题依然存在。
即然是从TC报的错误,就决定去TC看一下,果不其然,seata-service
报错如下:
2019-12-02 19:53:10,354 ERROR Failed to add branchSession to globalSession:Data truncation: Data too long for column 'lock_key' at row 1
io.seata.common.exception.StoreException: Data truncation: Data too long for column 'lock_key' at row 1
at io.seata.core.store.db.LogStoreDataBaseDAO.insertBranchTransactionDO(LogStoreDataBaseDAO.java:389)
at io.seata.server.store.db.DatabaseTransactionStoreManager.writeSession(DatabaseTransactionStoreManager.java:113)
at io.seata.server.session.db.DataBaseSessionManager.addBranchSession(DataBaseSessionManager.java:127)
at io.seata.server.session.AbstractSessionManager.onAddBranch(AbstractSessionManager.java:131)
at io.seata.server.session.GlobalSession.addBranch(GlobalSession.java:214)
at io.seata.server.coordinator.DefaultCore.lambda$branchRegister$0(DefaultCore.java:82)
at io.seata.server.session.GlobalSession.lockAndExcute(GlobalSession.java:594)
at io.seata.server.coordinator.DefaultCore.branchRegister(DefaultCore.java:66)
at io.seata.server.coordinator.DefaultCoordinator.doBranchRegister(DefaultCoordinator.java:186)
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:148)
at io.seata.server.AbstractTCInboundHandler$4.execute(AbstractTCInboundHandler.java:144)
at io.seata.core.exception.AbstractExceptionHandler.exceptionHandleTemplate(AbstractExceptionHandler.java:117)
at io.seata.server.AbstractTCInboundHandler.handle(AbstractTCInboundHandler.java:144)
at io.seata.core.protocol.transaction.BranchRegisterRequest.handle(BranchRegisterRequest.java:136)
at io.seata.server.coordinator.DefaultCoordinator.onRequest(DefaultCoordinator.java:473)
at io.seata.core.rpc.DefaultServerMessageListenerImpl.onTrxMessage(DefaultServerMessageListenerImpl.java:87)
at io.seata.core.rpc.netty.RpcServer.dispatch(RpcServer.java:266)
at io.seata.core.rpc.netty.AbstractRpcRemoting$3.run(AbstractRpcRemoting.java:371)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'lock_key' at row 1
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4230)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeUpdate(DruidPooledPreparedStatement.java:256)
at io.seata.core.store.db.LogStoreDataBaseDAO.insertBranchTransactionDO(LogStoreDataBaseDAO.java:387)
... 21 common frames omitted
问题解决
-
seata-service
的表branch_table
的lock_key
定义为varchar(128)
,批量修改时不能满足需求。将lock_key
修改为text
,问题解决。 -
翻看官方的GitHub,发现官方也于今天(2019.12.02)在
develop
版上修复了这个问题 feature: solve the lock_key length problem #1905,相信不久的release,就会解决这个问题。