……………………………………(大量相同的报错)…………………………………………
2017-08-31T11:11:04.291424Z32394522[ERROR]InnoDB:Recordinindex`t_idx`of table`$db_name`.`$tb_name`wasnotfound on update:TUPLE(info_bits=0,9fields):{[12]121098369601(0x010201000908030609060001),[9](0x000000000000010000),[4]KOWA(0x0B0F0701),[4]AYNA(0x01090E01),[6]STAT44(0x030401040404),[4]AYNA(0x01090E01),[1]0(0x00),[1]0(0x00),[32]8f2a39b44fe74cd781527d856342d834(0x0806020103090204040605070403040708010502070408050603040204080304)}at:COMPACT RECORD(info_bits=0,9fields):{[12]121098369601(0x010201000908030609060001),[9](0x000000000000010000),[4]KOWA(0x0B0F0701),[4]AYNA(0x01090E01),NULL,NULL,[1]0(0x00),[1]0(0x00),[32]8f2a39b44fe74cd781527d856342d834(0x0806020103090204040605070403040708010502070408050603040204080304)}
2017-08-31T03:11:04.291454Z32394522[Note]InnoDB:GIS MBR INFO:1.31506e-47and1.02964e-71,2.8816e-306,1.93059e+53
2017-08-3103:11:040x7fcaf04be700InnoDB:Assertionfailureinthread140509591627520infile row0ins.cc line282
InnoDB:Failingassertion:!cursor->index->is_committed()
InnoDB:Weintentionally generate a memory trap.
InnoDB:Submita detailed bug report to http://bugs.mysql.com.
InnoDB:Ifyougetrepeated assertion failuresorcrashes,even
InnoDB:immediately after the mysqld startup,there may be
InnoDB:corruptionintheInnoDBtablespace.Pleaserefer to
InnoDB:http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB:about forcing recovery.
03:11:04UTC-mysqld got signal6;
Thiscould be because you hit a bug.Itisalso possible thatthisbinary
orone of the libraries it was linked againstiscorrupt,improperly built,
ormisconfigured.Thiserror can also be causedbymalfunctioning hardware.
Attemptingto collect some information that could help diagnose the problem.
Asthisisa crashandsomethingisdefinitely wrong,the information
collection process might fail.
…………………………………………………………………………………………………………
Tryingtogetsome variables.
Somepointers may be invalidandcause thedumpto abort.
Query(7fca7c0dbaa0):isan invalid pointer
ConnectionID(thread ID):32394522
Status:NOT_KILLED
Themanual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you findoutwhatiscausing the crash.
…………………………………………………………………………………………………………
(重启中)
…………………………………………………………………………………………………………
2017-08-31T03:11:08.925622Z0[Note]$basedir/bin/mysqld:readyforconnections.
Version:'5.7.12-log'socket:'$datadir/mysqld.sock'port:3306Sourcedistribution
2017-08-31T03:31:10.232145Z1704[ERROR]InnoDB:Recordinindex`t_idx`of table`$db_name`.`$tb_name`wasnotfound on update:TUPLE(info_bits=0,9fields):{[12]198051077411(0x010908000501000707040101),[9]7(0x000000000000020700),[4]AOGA(0x010F0701),[4]AQGA(0x01010701),[6]STAT44(0x030401040404),NULL,[1]0(0x00),[1]0(0x00),[32]c6f98a358ace4897a11a27d689bb6884(0x0306060908010305080103050408090701010101020704060809020206080804)}at:COMPACT RECORD(info_bits=0,9fields):{[12]198051077411(0x010908000501000707040101),[9]7(0x000000000000020700),[4]AOGA(0x010F0701),[4]AQGA(0x01010701),NULL,NULL,[1]0(0x00),[1]0(0x00),[32]c6f98a358ace4897a11a27d689bb6884(0x0306060908010305080103050408090701010101020704060809020206080804)}
2017-08-31T03:31:10.232168Z1704[Note]InnoDB:GIS MBR INFO:7.26084e-43and1.08604e-42,2.8823e-306,132832
2017-08-31T03:35:51.201716Z2208[ERROR]InnoDB:Flaggedcorruption of`t_idx`intable`$db_name`.`$tb_name`inCHECK TABLE;Wrongcount
左右滑动查看代码
初步确定为因为名为 t_idx 的索引损坏导致的大量报错,并在处理 update 语句时导致 crash。
检查 binlog 发现的确有很多对该表的 update 操作。
执行一下 check table,发现的确有问题:
mysql>CHECK TABLE`$db_name`.`$tb_name`;
+--------------------+-------+----------+-------------------------------------------------------+
|Table|Op|Msg_type|Msg_text|
+--------------------+-------+----------+-------------------------------------------------------+
|$db_name.$tb_name|check|Warning|InnoDB:Indext_idxismarkedascorrupted|
|$db_name.$tb_name|check|error|Corrupt|
+--------------------+-------+----------+-------------------------------------------------------+
2rowsinset(0.83sec)
mysql>CHECK TABLE`$db_name`.`$tb_name`;
+--------------------+-------+----------+-------------------------------------------------------+
|Table|Op|Msg_type|Msg_text|
+--------------------+-------+----------+-------------------------------------------------------+
|$db_name.$tb_name|check|Warning|InnoDB:Indext_idxismarkedascorrupted|
|$db_name.$tb_name|check|error|Corrupt|
+--------------------+-------+----------+-------------------------------------------------------+
2rowsinset(0.83sec)
因该库为高可用主库,检查到备库状态正常,准备先手动做 failover,再对该表进行修复。
因为表小,也比较幸运,修复过程十分顺利:
mysql>OPTIMIZE TABLE`$db_name`.`$tb_name`;
+--------------------+----------+----------+-------------------------------------------------------------------+
|Table|Op|Msg_type|Msg_text|
+--------------------+----------+----------+-------------------------------------------------------------------+
|$db_name.$tb_name|optimize|note|Tabledoesnotsupport optimize,doing recreate+analyze instead|
|$db_name.$tb_name|optimize|status|OK|
+--------------------+----------+----------+-------------------------------------------------------------------+
2rowsinset(3.42sec)
mysql>ALTER TABLE`$db_name`.`$tb_name`ENGINE=INNODB;
QueryOK,0rows affected(3.09sec)
Records:0Duplicates:0Warnings:0
mysql>ANALYZE TABLE`$db_name`.`$tb_name`;
+--------------------+---------+----------+----------+
|Table|Op|Msg_type|Msg_text|
+--------------------+---------+----------+----------+
|$db_name.$tb_name|analyze|status|OK|
+--------------------+---------+----------+----------+
1rowinset(0.00sec)
mysql>CHECK TABLE`$db_name`.`$tb_name`;
+--------------------+-------+----------+----------+
|Table|Op|Msg_type|Msg_text|
+--------------------+-------+----------+----------+
|$db_name.$tb_name|check|status|OK|
+--------------------+-------+----------+----------+
1rowinset(0.98sec)
mysql>OPTIMIZE TABLE`$db_name`.`$tb_name`;
+--------------------+----------+----------+-------------------------------------------------------------------+
|Table|Op|Msg_type|Msg_text|
+--------------------+----------+----------+-------------------------------------------------------------------+
|$db_name.$tb_name|optimize|note|Tabledoesnotsupport optimize,doing recreate+analyze instead|
|$db_name.$tb_name|optimize|status|OK|
+--------------------+----------+----------+-------------------------------------------------------------------+
2rowsinset(3.42sec)
mysql>ALTER TABLE`$db_name`.`$tb_name`ENGINE=INNODB;
QueryOK,0rows affected(3.09sec)
Records:0Duplicates:0Warnings:0
mysql>ANALYZE TABLE`$db_name`.`$tb_name`;
+--------------------+---------+----------+----------+
|Table|Op|Msg_type|Msg_text|
+--------------------+---------+----------+----------+
|$db_name.$tb_name|analyze|status|OK|
+--------------------+---------+----------+----------+
1rowinset(0.00sec)
mysql>CHECK TABLE`$db_name`.`$tb_name`;
+--------------------+-------+----------+----------+
|Table|Op|Msg_type|Msg_text|
+--------------------+-------+----------+----------+
|$db_name.$tb_name|check|status|OK|
+--------------------+-------+----------+----------+
1rowinset(0.98sec)
参考文档:
关于 mysqlcheck 与 check/analyze/optimize table 等命令:
http://blog.itpub.net/29773961/viewspace-1815688/
比较类似的一个 case 被提到了 bug 库:
https://bugs.mysql.com/bug.php?id=82997
问题描述节选:
Withsome random DML running I managed to hit a problem on5.7.13.
Nextstepformeisto test current versionandmake a suitable testcase.
Version:'5.7.13'socket:''port:3306MySQLCommunityServer(GPL)
[ERROR]InnoDB:Recordinindex`ed`of table`test`.`users`was n
[Note]InnoDB:GIS MBR INFO:1.20768e-153and4.76881e-038,7.0436
InnoDB:Assertionfailureinthread2384infile row0ins.cc line282
InnoDB:Failingassertion:!cursor->index->is_committed()
Withsome random DML running I managed to hit a problem on5.7.13.
Nextstepformeisto test current versionandmake a suitable testcase.
Version:'5.7.13'socket:''port:3306MySQLCommunityServer(GPL)
[ERROR]InnoDB:Recordinindex`ed`of table`test`.`users`was n
[Note]InnoDB:GIS MBR INFO:1.20768e-153and4.76881e-038,7.0436
InnoDB:Assertionfailureinthread2384infile row0ins.cc line282
InnoDB:Failingassertion:!cursor->index->is_committed()
SACC2017
即日起,凡填写 SACC2017 会前调查问卷【点击阅读原文即可参与~】,即有机会赢取价值 2600 元的大会门票 1 张!仅限 10 张!