Could not found global transaction xid

项目场景：

生产环境使用 SEATA 执行分布式事务异常

---

问题描述

订单服务提交 seata 分布式事务时报错

org.springframework.jdbc.UncategorizedSQLException: 
### Error updating database.  Cause: java.sql.SQLException: io.seata.core.exception.RmTransactionException: Response[ TransactionException[Could not found global transaction xid = 172.19.
100.11:8091:4134702464872835595, may be has finished.] ]
### The error may exist in cn/xxx/pay/biz/payment/mapper/PaymentMapper.java (best guess)
### The error may involve cn.xxx.pay.biz.payment.mapper.PaymentMapper.updateById-Inline
### The error occurred while setting parameters
### SQL: UPDATE payments  SET transaction_id=?,  status=?,      updatedAt=?,    finished_time=?, fee=?  WHERE id=?
### Cause: java.sql.SQLException: io.seata.core.exception.RmTransactionException: Response[ TransactionException[Could not found global transaction xid = 172.19.100.11:8091:41347024648728
35595, may be has finished.] ]
; uncategorized SQLException; SQL state [null]; error code [0]; io.seata.core.exception.RmTransactionException: Response[ TransactionException[Could not found global transaction xid = 172
.19.100.11:8091:4134702464872835595, may be has finished.] ]
	at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:92)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:441)
	at jdk.proxy2/jdk.proxy2.$Proxy176.update(Unknown Source)
	at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:288)
	at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:64)
	at com.baomidou.mybatisplus.core.override.MybatisMapperProxy$PlainMethodInvoker.invoke(MybatisMapperProxy.java:148)
	at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:89)
	at jdk.proxy2/jdk.proxy2.$Proxy188.updateById(Unknown Source)
	at com.baomidou.mybatisplus.extension.service.IService.updateById(IService.java:239)
	at cn.xxx.pay.biz.payment.service.impl.PaymentNotifyServiceImpl.updatePayment(PaymentNotifyServiceImpl.java:340)
	at cn.xxx.pay.biz.payment.service.impl.PaymentNotifyServiceImpl.paySuccess(PaymentNotifyServiceImpl.java:205)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
	at cn.xxx.pay.biz.payment.service.impl.PaymentNotifyServiceImpl$$SpringCGLIB$$0.paySuccess(<generated>)
	at cn.xxx.pay.biz.mq.listener.PaymentQueryListener.dealOrderPayNotify(PaymentQueryListener.java:71)
	at jdk.internal.reflect.GeneratedMethodAccessor560.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
	at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:75)
	at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:263)
	at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandlerAndProcessResult(MessagingMessageListenerAdapter.java:210)
	at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:149)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1718)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1637)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1625)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1616)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListenerAndHandleException(AbstractMessageListenerContainer.java:1561)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.lambda$executeListener$8(AbstractMessageListenerContainer.java:1540)
	at io.micrometer.observation.Observation.observe(Observation.java:492)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1540)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1001)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:948)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1326)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1232)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.sql.SQLException: io.seata.core.exception.RmTransactionException: Response[ TransactionException[Could not found global transaction xid = 172.19.100.11:8091:41347024648728
35595, may be has finished.] ]
	at io.seata.rm.datasource.ConnectionProxy.recognizeLockKeyConflictException(ConnectionProxy.java:161)
	at io.seata.rm.datasource.ConnectionProxy.processGlobalTransactionCommit(ConnectionProxy.java:252)
	at io.seata.rm.datasource.ConnectionProxy.doCommit(ConnectionProxy.java:230)
	at io.seata.rm.datasource.ConnectionProxy.lambda$commit$0(ConnectionProxy.java:188)
	at io.seata.rm.datasource.ConnectionProxy$LockRetryPolicy.execute(ConnectionProxy.java:344)
	at io.seata.rm.datasource.ConnectionProxy.commit(ConnectionProxy.java:187)
	at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor.lambda$executeAutoCommitTrue$2(AbstractDMLBaseExecutor.java:138)
	at io.seata.rm.datasource.ConnectionProxy$LockRetryPolicy.doRetryOnLockConflict(ConnectionProxy.java:356)
	at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor$LockRetryPolicy.execute(AbstractDMLBaseExecutor.java:180)
	at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor.executeAutoCommitTrue(AbstractDMLBaseExecutor.java:136)
	at io.seata.rm.datasource.exec.AbstractDMLBaseExecutor.doExecute(AbstractDMLBaseExecutor.java:82)
	at io.seata.rm.datasource.exec.BaseTransactionalExecutor.execute(BaseTransactionalExecutor.java:125)
	at io.seata.rm.datasource.exec.ExecuteTemplate.execute(ExecuteTemplate.java:137)
	at io.seata.rm.datasource.exec.ExecuteTemplate.execute(ExecuteTemplate.java:56)
	at io.seata.rm.datasource.PreparedStatementProxy.execute(PreparedStatementProxy.java:55)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
	at jdk.internal.reflect.GeneratedMethodAccessor169.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:64)
	at jdk.proxy2/jdk.proxy2.$Proxy231.update(Unknown Source)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
	at jdk.internal.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49)
	at com.baomidou.mybatisplus.extension.plugins.MybatisPlusInterceptor.intercept(MybatisPlusInterceptor.java:106)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:62)
	at jdk.proxy2/jdk.proxy2.$Proxy230.update(Unknown Source)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194)
	at jdk.internal.reflect.GeneratedMethodAccessor227.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
	... 39 common frames omitted
Caused by: io.seata.core.exception.RmTransactionException: Response[ TransactionException[Could not found global transaction xid = 172.19.100.11:8091:4134702464872835595, may be has finis
hed.] ]
	at io.seata.rm.AbstractResourceManager.branchRegister(AbstractResourceManager.java:69)
	at io.seata.rm.DefaultResourceManager.branchRegister(DefaultResourceManager.java:96)
	at io.seata.rm.datasource.ConnectionProxy.register(ConnectionProxy.java:273)
	at io.seata.rm.datasource.ConnectionProxy.processGlobalTransactionCommit(ConnectionProxy.java:250)
	... 73 common frames omitted

---

原因分析：

1、seta 跨服务 xid 传播时会将 xid 记录在线程的 ThredLocal 中（RootContext 管理）
2、如果当前线程完成当前任务时未清除 ThredLocal 中的 xid （如：未 commit/rollback）
3、当下一个任务到来，该线程执行下一个分布式事务时将尝试完成上一个分布式事务
4、此时如果上一个分布式事务超时，则 seata server 会回滚超时的事务并删除 seata server 中的记录
5、当前线程提交上一个事务时 seata server 中无此 xid 出现上述异常

---

解决方案：

使用 GlobalTransaction 编程式事务时，一定要确保方法结束/异常时 commit/rollback