错误信息:
2020-08-13 14:13:04.179 ERROR 393116 --- [nio-9099-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.CannotAcquireLockException:
### Error updating database. Cause: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may involve com.caixun.dal.master_data_source.match.mybatis.SoccerMatchMapper.updateByPrimaryKeySelective-Inline
### The error occurred while setting parameters
### SQL: update soccer_matchs SET syn_match_id = ?, syn_league_id = ?, syn_league_name = ?, hometeam_id = ?, hometeam_name = ?, visitorteam_id = ?, visitorteam_name = ?, `time` = ?, status = ?, first_half_score = ?, second_half_score = ?, score = ?, subindex_initial = ?, subindex_spot = ?, rqspf_initial = ?, rqspf_spot = ?, euroindex_initial = ?, euroindex_spot = ?, bigsmall_initial = ?, bigsmall_spot = ?, lottery_id = ?, match_date = ?, match_status = ?, march_show_status = ?, created_at = ?, updated_at = ? where match_id = ?
### Cause: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; ]; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction] with root cause
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:121)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:960)
at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:388)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:46)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
at com.sun.proxy.$Proxy125.update(Unknown Source)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:198)
at sun.reflect.GeneratedMethodAccessor149.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
at com.sun.proxy.$Proxy62.update(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:294)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:63)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59)
at com.sun.proxy.$Proxy65.updateByPrimaryKeySelective(Unknown Source)
at com.caixun.biz.dataApi.service.serviceImpl.ApiMatchServiceImpl.inputSoccerMatch(ApiMatchServiceImpl.java:131)
at com.caixun.biz.dataApi.service.serviceImpl.ApiMatchServiceImpl.inputSoccerMatchInfo(ApiMatchServiceImpl.java:72)
at com.caixun.biz.dataApi.service.serviceImpl.ApiMatchServiceImpl$$FastClassBySpringCGLIB$$b20824a4.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.caixun.biz.dataApi.service.serviceImpl.ApiMatchServiceImpl$$EnhancerBySpringCGLIB$$3027383c.inputSoccerMatchInfo(<generated>)
at com.caixun.web.controller.dataAPI.MatchController.soccerMatch(MatchController.java:42)
at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
原因分析:
在高并发的情况下,Spring事物造成数据库死锁,后续操作超时抛出异常。
Mysql数据库采用InnoDB模式,默认参数:innodb_lock_wait_timeout设置锁等待的时间是50s,一旦数据库锁超过这个时间就会报错。
解决方案:
一、通过下面语句查找到为提交事务的数据,kill掉此线程即可。
#当务之急,也是要看看数据库中有没有比较长时间执行的sql:
show processlist;
#当前所运行的所有事务
SELECT * FROM information_schema.INNODB_TRX;
#当前所有的锁
#SELECT * FROM information_schema.INNODB_LOCKs;
#锁等待的对应的关系
#SELECT * FROM information_schema.INNODB_LOCK_waits;
那么我们看到事务表中INNODB_TRX,里面是否有正在锁定的事务线程,看看ID是否在show processlist里面的sleep线程中,如果有,那么就证明了这个休眠的线程事务一直没有commit(提交)或者roolback(回滚)而是卡住了,所以,我们需要人为介入,kill掉。
如果发现了好多事务任务,那最好都kill掉。
命令为
select concat('KILL ',id,';') from information_schema.processlist where user='cms_bokong';
通过information_schema.processlist表中的连接信息生成需要处理掉的Mysql连接的语句临时文件,然后执行文件中生成的指令。然后我们获取到了对应任务的id,一个一个 kill id就行了。
然后我们再去找还在进行事务的任务,就会发现空掉了。
主要看着几个字段,如果有阻塞数据就不是0,找到后在根据下图这个字段杀死进程
kill id ;(杀死对应id的进程)
前面已经了解了InnoDB关于在出现锁等待的时候,会根据参数innodb_lock_wait_timeout的配置,判断是否需要进行timeout的操作,本文档介绍在出现锁等待时候的查看及分析处理;
在InnoDB Plugin之前,一般通过show full processlist(很难发现被锁的行记录问题所在)和show engine innodb status命令查看当前的数据库请求,然后再判断当前事务中锁的情况。随着mysql的发展,已经提供更加便捷的方法来监控数据库中的锁等待现象了。
在information_schema下面有三张表:INNODB_TRX、INNODB_LOCKS、INNODB_LOCK_WAITS(解决问题方法),通过这三张表,可以更简单地监控当前的事务并分析可能存在的问题。
比较常用的列:
trx_id:InnoDB存储引擎内部唯一的事物ID
trx_status:当前事务的状态
trx_status:事务的开始时间
trx_requested_lock_id:等待事务的锁ID
trx_wait_started:事务等待的开始时间
trx_weight:事务的权重,反应一个事务修改和锁定的行数,当发现死锁需要回滚时,权重越小的值被回滚
trx_mysql_thread_id:MySQL中的进程ID,与show processlist中的ID值相对应
trx_query:事务运行的SQL语句
工作中遇到了一条车辆记录,select * from car for update 或者是修改某个字段的值,就报错:Lock wait timeout exceeded; try restarting transaction解决方法,另外从根本上从业务逻辑代码优化对数据库的操作,之前也遇到过此类情况,比如刚刚修改完这条记录,接着再次修改,也会报此错误,从代码和业务层面尽量避免开来。
二、增加锁等待时间,即增大下面配置项参数值,单位为秒(s)
innodb_lock_wait_timeout=500
#查询全局等待事务锁超时时间
SHOW GLOBAL VARIABLES LIKE 'innodb_lock_wait_timeout';
#设置全局等待事务锁超时时间
SET GLOBAL innodb_lock_wait_timeout=500;
#查询当前会话等待事务锁超时时间
SHOW VARIABLES LIKE 'innodb_lock_wait_timeout';
应急处理完成后,我们就需要核对原因,防止以后再出现:
1.mysql的引擎检查,可以检查一下数据库引擎是不是InnoDB(mysql5.5.5以前默认是MyISAM,Mysql5.5.5以后默认是InnoDB),show engines;#检查命令
如果不是INNDB,那么就改为InnDB;
命令为:
#查看表使用的存储引擎
show table status from db_name where name='table_name';
#修改表的存储引擎
alter table table_name engine=innodb;
三、
优化存储过程,事务避免过长时间的等待。
参考信息
1、锁等待超时。是当前事务在等待其它事务释放锁资源造成的。可以找出锁资源竞争的表和语句,优化SQL,创建索引等。如果还是不行,可以适当减少并发线程数。
2、事务在等待给某个表加锁时超时,估计是表正被另的进程锁住一直没有释放。
可以用 SHOW INNODB STATUS/G; 看一下锁的情况。
3、搜索解决之道,在管理节点的[ndbd default]区加:
TransactionDeadLockDetectionTimeOut=10000(设置 为10秒)默认是1200(1.2秒)
4、InnoDB会自动的检测死锁进行回滚,或者终止死锁的情况。
InnoDB automatically detects transaction deadlocks and rolls back a transaction or transactions to break the deadlock. InnoDB tries to pick small transactions to roll back, where the size of a transaction is determined by the number of rows inserted, updated, or deleted.
如果参数innodb_table_locks=1并且autocommit=0时,InnoDB会留意表的死锁,和MySQL层面的行级锁。另外,InnoDB不会检测MySQL的Lock Tables命令和其他存储引擎死锁。你应该设置innodb_lock_wait_timeout来解决这种情况。
innodb_lock_wait_timeout是Innodb放弃行级锁的超时时间。