mysql get error 28_MySQL Got error 134错误排查

现象

Discuz论坛一张表反复崩溃,导致帖子页无法打开,日志如下:

150729 23:24:17 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect key file for table './bbs/forum_threadaddviews.MYI'; try to repair it

150729 23:24:18 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect key file for table './bbs/forum_threadaddviews.MYI'; try to repair it

150729 23:24:24 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect key file for table './bbs/forum_threadaddviews.MYI'; try to repair it

150729 23:24:33 [ERROR] Got error 134 when reading table './bbs/forum_forumrecommend'

150729 23:24:36 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect key file for table './bbs/forum_threadaddviews.MYI'; try to repair it

...

150729 23:25:51 [ERROR] /usr/local/mysql/libexec/mysqld: Table './bbs/forum_threadaddviews' is marked as crashed and should be repaired

150729 23:25:53 [ERROR] /usr/local/mysql/libexec/mysqld: Table './bbs/forum_threadaddviews' is marked as crashed and should be repaired

150729 23:25:53 [ERROR] /usr/local/mysql/libexec/mysqld: Table './bbs/forum_threadaddviews' is marked as crashed and should be repaired

150729 23:25:55 [ERROR] Got error 134 when reading table './bbs/forum_forumrecommend'

150729 23:25:55 [ERROR] /usr/local/mysql/libexec/mysqld: Table './bbs/forum_threadaddviews' is marked as crashed and should be repaired

150729 23:25:56 [ERROR] /usr/local/mysql/libexec/mysqld: Table './bbs/forum_threadaddviews' is marked as crashed and should be repaired

150729 23:25:56 [ERROR] /usr/local/mysql/libexec/mysqld: Table './bbs/forum_threadaddviews' is marked as crashed and should be repaired

1

2

3

4

5

6

7

8

9

10

11

12

13

15072923:24:17[ERROR]/usr/local/mysql/libexec/mysqld:Incorrectkeyfilefortable'./bbs/forum_threadaddviews.MYI';trytorepairit

15072923:24:18[ERROR]/usr/local/mysql/libexec/mysqld:Incorrectkeyfilefortable'./bbs/forum_threadaddviews.MYI';trytorepairit

15072923:24:24[ERROR]/usr/local/mysql/libexec/mysqld:Incorrectkeyfilefortable'./bbs/forum_threadaddviews.MYI';trytorepairit

15072923:24:33[ERROR]Goterror134whenreadingtable'./bbs/forum_forumrecommend'

15072923:24:36[ERROR]/usr/local/mysql/libexec/mysqld:Incorrectkeyfilefortable'./bbs/forum_threadaddviews.MYI';trytorepairit

...

15072923:25:51[ERROR]/usr/local/mysql/libexec/mysqld:Table'./bbs/forum_threadaddviews'ismarkedascrashedandshouldberepaired

15072923:25:53[ERROR]/usr/local/mysql/libexec/mysqld:Table'./bbs/forum_threadaddviews'ismarkedascrashedandshouldberepaired

15072923:25:53[ERROR]/usr/local/mysql/libexec/mysqld:Table'./bbs/forum_threadaddviews'ismarkedascrashedandshouldberepaired

15072923:25:55[ERROR]Goterror134whenreadingtable'./bbs/forum_forumrecommend'

15072923:25:55[ERROR]/usr/local/mysql/libexec/mysqld:Table'./bbs/forum_threadaddviews'ismarkedascrashedandshouldberepaired

15072923:25:56[ERROR]/usr/local/mysql/libexec/mysqld:Table'./bbs/forum_threadaddviews'ismarkedascrashedandshouldberepaired

15072923:25:56[ERROR]/usr/local/mysql/libexec/mysqld:Table'./bbs/forum_threadaddviews'ismarkedascrashedandshouldberepaired

表forum_threadaddviews修复后,mysql日志仍然在刷ERROR错误:

[root@localhost log]# tail -f mysql.err

150730 0:03:17 [ERROR] Got error 134 when reading table './bbs/forum_forumrecommend'

150730 0:05:29 [ERROR] Got error 134 when reading table './bbs/forum_forumrecommend'

150730 0:05:45 [ERROR] Got error 134 when reading table './bbs/forum_forumrecommend'

150730 0:06:34 [ERROR] Got error 134 when reading table './bbs/forum_forumrecommend'

150730 0:06:49 [ERROR] Got error 134 when reading table './bbs/forum_forumrecommend'

1

2

3

4

5

6

[root@localhostlog]# tail -f mysql.err

1507300:03:17[ERROR]Goterror134whenreadingtable'./bbs/forum_forumrecommend'

1507300:05:29[ERROR]Goterror134whenreadingtable'./bbs/forum_forumrecommend'

1507300:05:45[ERROR]Goterror134whenreadingtable'./bbs/forum_forumrecommend'

1507300:06:34[ERROR]Goterror134whenreadingtable'./bbs/forum_forumrecommend'

1507300:06:49[ERROR]Goterror134whenreadingtable'./bbs/forum_forumrecommend'

追查

通过MySQL错误日志,可以看到Got error 134在13年8月和15年7月集中爆发,符合论坛近期的表现

[root@localhost var]# grep "Got error 134" mysql.err |awk '{print $1}' |sort |uniq -c

3450 130112

213 130801

325 130802

266 130803

214 130804

216 130805

212 130806

271 130807

178 130808

170 130809

188 130810

194 130811

120 130812

136 130813

207 130814

185 130815

160 130816

181 130817

137 130818

1 150706

749 150713

1333 150714

1551 150715

1569 150716

1569 150717

1522 150718

1557 150719

1534 150720

1508 150721

1430 150722

1406 150723

1400 150724

1414 150725

1469 150726

1343 150727

1360 150728

1529 150729

44 150730

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

[root@localhostvar]# grep "Got error 134" mysql.err |awk '{print $1}' |sort |uniq -c

3450130112

213130801

325130802

266130803

214130804

216130805

212130806

271130807

178130808

170130809

188130810

194130811

120130812

136130813

207130814

185130815

160130816

181130817

137130818

1150706

749150713

1333150714

1551150715

1569150716

1569150717

1522150718

1557150719

1534150720

1508150721

1430150722

1406150723

1400150724

1414150725

1469150726

1343150727

1360150728

1529150729

44150730

Got error 134的表,几乎都是 forum_forumrecommend

[root@localhost var]# grep "Got error 134" mysql.err | grep "^1507" |awk '{print $NF}' |sort |uniq -c

24287 './bbs/forum_forumrecommend'

1 './bbs/forum_threadaddviews'

1

2

3

[root@localhostvar]# grep "Got error 134" mysql.err | grep "^1507" |awk '{print $NF}' |sort |uniq -c

24287'./bbs/forum_forumrecommend'

1'./bbs/forum_threadaddviews'

forum_threadaddviews表情况,也差不多是13年8月和15年7月比较集中

[root@localhost var]# grep "forum_threadaddviews" mysql.err |awk '{print $1}' |sort |uniq -c

8312 121222

4857 130101

4874 130112

12810 130126

264 130128

4260 130129

2873 130219

5507 130222

19385 130223

6753 130224

213 130801

325 130802

266 130803

214 130804

216 130805

212 130806

271 130807

178 130808

170 130809

188 130810

194 130811

120 130812

136 130813

207 130814

185 130815

160 130816

181 130817

137 130818

34149 150311

16083 150312

3944 150415

5343 150515

1491 150516

21939 150519

557 150625

26084 150626

22901 150629

36339 150706

84450 150713

54706 150714

43947 150715

1086 150717

42033 150718

52414 150719

77772 150720

70164 150721

73334 150723

41844 150724

40190 150725

65812 150726

27799 150727

39230 150728

23982 150729

[root@localhost var]#

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

[root@localhostvar]# grep "forum_threadaddviews" mysql.err |awk '{print $1}' |sort |uniq -c

8312121222

4857130101

4874130112

12810130126

264130128

4260130129

2873130219

5507130222

19385130223

6753130224

213130801

325130802

266130803

214130804

216130805

212130806

271130807

178130808

170130809

188130810

194130811

120130812

136130813

207130814

185130815

160130816

181130817

137130818

34149150311

16083150312

3944150415

5343150515

1491150516

21939150519

557150625

26084150626

22901150629

36339150706

84450150713

54706150714

43947150715

1086150717

42033150718

52414150719

77772150720

70164150721

73334150723

41844150724

40190150725

65812150726

27799150727

39230150728

23982150729

[root@localhostvar]#

Got error 134

Even though the MyISAM table format is very reliable (all changes to a table made by an SQL statement are written before the statement returns), you can still get corrupted tables if any of the following events occur:

The mysqld process is killed in the middle of a write.

An unexpected computer shutdown occurs (for example, the computer is turned off).

Hardware failures.

You are using an external program (such as myisamchk ) to modify a table that is being modified by the server at the same time.

A software bug in the MySQL or MyISAM code.

Typical symptoms of a corrupt table are: You get the following error while selecting data from the table:

Incorrect key file for table: '...'. Try to repair it

Queries don't find rows in the table or return incomplete results.

来自:http://dev.mysql.com/doc/refman/5.0/en/corrupted-myisam-tables.html

解决:

CHECK TABLE 表名

REPAIR TABLE 表名

1

2

CHECKTABLE表名

REPAIRTABLE表名

修复

检查forum_forumrecommend,可以看到确实有错误

mysql> check table forum_forumrecommend;

+-----------------------------+-------+----------+-----------------------------------------------------------+

| Table | Op | Msg_type | Msg_text |

+-----------------------------+-------+----------+-----------------------------------------------------------+

| bbs.forum_forumrecommend | check | warning | 70 clients are using or haven't closed the table properly |

| bbs.forum_forumrecommend | check | error | Record-count is not ok; is 192 Should be: 193 |

| bbs.forum_forumrecommend | check | warning | Found 415 deleted space. Should be 0 |

| bbs.forum_forumrecommend | check | warning | Found 1 deleted blocks Should be: 0 |

| bbs.forum_forumrecommend | check | error | Corrupt |

+-----------------------------+-------+----------+-----------------------------------------------------------+

1

2

3

4

5

6

7

8

9

10

mysql>checktableforum_forumrecommend;

+-----------------------------+-------+----------+-----------------------------------------------------------+

|Table|Op|Msg_type|Msg_text|

+-----------------------------+-------+----------+-----------------------------------------------------------+

|bbs.forum_forumrecommend|check|warning|70clientsareusingorhaven'tclosedthetableproperly|

|bbs.forum_forumrecommend|check|error|Record-countisnotok;is192Shouldbe:193|

|bbs.forum_forumrecommend|check|warning|Found415deletedspace.Shouldbe0|

|bbs.forum_forumrecommend|check|warning|Found1deletedblocksShouldbe:0|

|bbs.forum_forumrecommend|check|error|Corrupt|

+-----------------------------+-------+----------+-----------------------------------------------------------+

修复forum_forumrecommend

mysql> repair table forum_forumrecommend;

+-----------------------------+--------+----------+----------------------------------------+

| Table | Op | Msg_type | Msg_text |

+-----------------------------+--------+----------+----------------------------------------+

| bbs.forum_forumrecommend | repair | warning | Number of rows changed from 193 to 192 |

| bbs.forum_forumrecommend | repair | status | OK |

+-----------------------------+--------+----------+----------------------------------------+

2 rows in set (0.01 sec)

mysql> check table forum_forumrecommend;

+-----------------------------+-------+----------+----------+

| Table | Op | Msg_type | Msg_text |

+-----------------------------+-------+----------+----------+

| bbs.forum_forumrecommend | check | status | OK |

+-----------------------------+-------+----------+----------+

1 row in set (0.00 sec)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

mysql>repairtableforum_forumrecommend;

+-----------------------------+--------+----------+----------------------------------------+

|Table|Op|Msg_type|Msg_text|

+-----------------------------+--------+----------+----------------------------------------+

|bbs.forum_forumrecommend|repair|warning|Numberofrowschangedfrom193to192|

|bbs.forum_forumrecommend|repair|status|OK|

+-----------------------------+--------+----------+----------------------------------------+

2rowsinset(0.01sec)

mysql>checktableforum_forumrecommend;

+-----------------------------+-------+----------+----------+

|Table|Op|Msg_type|Msg_text|

+-----------------------------+-------+----------+----------+

|bbs.forum_forumrecommend|check|status|OK|

+-----------------------------+-------+----------+----------+

1rowinset(0.00sec)

再次追查

看message,时间与上述问题比较吻合,因此可能是系统oom killer杀掉mysqld进程导致上述问题。

[root@localhost log]# grep -i "out of mem" messages* |grep mysqld

messages-20150705:Jun 29 04:21:21 localhost kernel: Out of memory: Kill process 14957 (mysqld) score 118 or sacrifice child

messages-20150705:Jun 29 04:29:02 localhost kernel: Out of memory: Kill process 4169 (mysqld) score 35 or sacrifice child

messages-20150705:Jun 29 21:17:52 localhost kernel: Out of memory: Kill process 5172 (mysqld) score 78 or sacrifice child

messages-20150705:Jun 29 21:21:36 localhost kernel: Out of memory: Kill process 14353 (mysqld) score 32 or sacrifice child

messages-20150705:Jun 29 21:21:39 localhost kernel: Out of memory: Kill process 14567 (mysqld) score 32 or sacrifice child

messages-20150705:Jun 30 01:44:37 localhost kernel: Out of memory: Kill process 15232 (mysqld) score 74 or sacrifice child

messages-20150705:Jul 1 00:11:28 localhost kernel: Out of memory: Kill process 31564 (mysqld) score 90 or sacrifice child

messages-20150705:Jul 1 17:32:53 localhost kernel: Out of memory: Kill process 9870 (mysqld) score 86 or sacrifice child

messages-20150712:Jul 7 01:23:53 localhost kernel: Out of memory: Kill process 18756 (mysqld) score 91 or sacrifice child

messages-20150712:Jul 7 01:23:54 localhost kernel: Out of memory: Kill process 3854 (mysqld) score 91 or sacrifice child

messages-20150712:Jul 11 01:06:10 localhost kernel: Out of memory: Kill process 4237 (mysqld) score 110 or sacrifice child

messages-20150712:Jul 11 01:06:10 localhost kernel: Out of memory: Kill process 16005 (mysqld) score 110 or sacrifice child

messages-20150719:Jul 13 10:19:33 localhost kernel: Out of memory: Kill process 16306 (mysqld) score 121 or sacrifice child

messages-20150726:Jul 22 21:58:46 localhost kernel: Out of memory: Kill process 29585 (mysqld) score 107 or sacrifice child

messages-20150726:Jul 22 22:12:49 localhost kernel: Out of memory: Kill process 10021 (mysqld) score 50 or sacrifice child

messages-20150726:Jul 24 16:51:24 localhost kernel: Out of memory: Kill process 12025 (mysqld) score 95 or sacrifice child

messages-20150726:Jul 24 16:51:24 localhost kernel: Out of memory: Kill process 21597 (mysqld) score 95 or sacrifice child

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

[root@localhostlog]# grep -i "out of mem" messages* |grep mysqld

messages-20150705:Jun2904:21:21localhostkernel:Outofmemory:Killprocess14957(mysqld)score118orsacrificechild

messages-20150705:Jun2904:29:02localhostkernel:Outofmemory:Killprocess4169(mysqld)score35orsacrificechild

messages-20150705:Jun2921:17:52localhostkernel:Outofmemory:Killprocess5172(mysqld)score78orsacrificechild

messages-20150705:Jun2921:21:36localhostkernel:Outofmemory:Killprocess14353(mysqld)score32orsacrificechild

messages-20150705:Jun2921:21:39localhostkernel:Outofmemory:Killprocess14567(mysqld)score32orsacrificechild

messages-20150705:Jun3001:44:37localhostkernel:Outofmemory:Killprocess15232(mysqld)score74orsacrificechild

messages-20150705:Jul100:11:28localhostkernel:Outofmemory:Killprocess31564(mysqld)score90orsacrificechild

messages-20150705:Jul117:32:53localhostkernel:Outofmemory:Killprocess9870(mysqld)score86orsacrificechild

messages-20150712:Jul701:23:53localhostkernel:Outofmemory:Killprocess18756(mysqld)score91orsacrificechild

messages-20150712:Jul701:23:54localhostkernel:Outofmemory:Killprocess3854(mysqld)score91orsacrificechild

messages-20150712:Jul1101:06:10localhostkernel:Outofmemory:Killprocess4237(mysqld)score110orsacrificechild

messages-20150712:Jul1101:06:10localhostkernel:Outofmemory:Killprocess16005(mysqld)score110orsacrificechild

messages-20150719:Jul1310:19:33localhostkernel:Outofmemory:Killprocess16306(mysqld)score121orsacrificechild

messages-20150726:Jul2221:58:46localhostkernel:Outofmemory:Killprocess29585(mysqld)score107orsacrificechild

messages-20150726:Jul2222:12:49localhostkernel:Outofmemory:Killprocess10021(mysqld)score50orsacrificechild

messages-20150726:Jul2416:51:24localhostkernel:Outofmemory:Killprocess12025(mysqld)score95orsacrificechild

messages-20150726:Jul2416:51:24localhostkernel:Outofmemory:Killprocess21597(mysqld)score95orsacrificechild

参考资料

[1]. http://blog.sina.com.cn/s/blog_4550f3ca0100x7kf.html

[2]. http://blog.csdn.net/hjue/article/details/1957256

1

2

[1].http://blog.sina.com.cn/s/blog_4550f3ca0100x7kf.html

[2].http://blog.csdn.net/hjue/article/details/1957256

博客能带货吗

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值