linux kill mysql_服务器内存不足导致LINUX KILL MYSQLD 故障

环境说明

OS操作系统：SUSE Linux 11

数据库版 本：MYSQL 5.6

故障问题描述

客户反映MYSQL数据库因为内存不足被重启。

故障分析

1、根据MYSQL报警日志确认问题：

2016-10-24 12:28:49 29226 [Note] Plugin

'FEDERATED' is disabled.

2016-10-24 12:28:49 29226 [Note] InnoDB:

Using atomics to ref count buffer pool pages

2016-10-24 12:28:49 29226 [Note] InnoDB: The

InnoDB memory heap is disabled

2016-10-24 12:28:49 29226 [Note] InnoDB:

Mutexes and rw_locks use GCC atomic builtins

2016-10-24 12:28:49 29226 [Note] InnoDB:

Compressed tables use zlib 1.2.3

2016-10-24 12:28:49 29226 [Note] InnoDB:

Using Linux native AIO

2016-10-24 12:28:49 29226 [Note] InnoDB:

Using CPU crc32 instructions

2016-10-24 12:28:49 29226 [Note] InnoDB:

Initializing buffer pool, size = 2.0G

2016-10-24 12:28:50 29226 [Note] InnoDB:

Completed initialization of buffer pool

2016-10-24 12:28:50 29226 [Note] InnoDB:

Highest supported file format is Barracuda.

2016-10-24 12:28:50 29226 [Note] InnoDB: Log

scan progressed past the checkpoint lsn 27852365485

2016-10-24 12:28:50 29226 [Note] InnoDB:

Database was not shutdown normally!

2016-10-24 12:28:50 29226 [Note] InnoDB:

Starting crash recovery.

2016-10-24 12:28:50 29226 [Note] InnoDB:

Reading tablespace information from the .ibd files...

2016-10-24 12:28:50 29226 [Note] InnoDB:

Restoring possible half-written data pages

2016-10-24 12:28:50 29226 [Note] InnoDB:

from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to log

sequence number 27852370215

2016-10-24 12:28:51 29226 [Note] InnoDB:

Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 13 14 15 16 17 18

19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

97 98 99

InnoDB: Apply batch completed

InnoDB: Last MySQL binlog file position 0

401322767, file name TSPTONLDB01-it_mgmad-bin.000023

2016-10-24 12:28:52 29226 [Note] InnoDB: 128

rollback segment(s) are active.

2016-10-24 12:28:52 29226 [Note] InnoDB:

5.6.19 started; log sequence number 27852370215

2016-10-24 12:28:52 29226 [Note] Recovering

after a crash using /dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin

2016-10-24 12:29:00 29226 [Note] Starting

crash recovery...

2016-10-24 12:29:00 29226 [Note] Crash

recovery finished.

2016-10-24 12:29:06 29226 [Note] Server

hostname (bind-address): '*'; port: 60002

2016-10-24 12:29:06 29226 [Note] IPv6 is

available.

2016-10-24 12:29:06 29226 [Note]- '::' resolves to '::';

2016-10-24 12:29:06 29226 [Note] Server

socket created on IP: '::'.

2016-10-24 12:29:06 29226 [Warning]

'proxies_priv' entry '@ root@tsptonldb01' ignored in --skip-name-resolve mode.

2016-10-24 12:29:06 29226 [Note] Event

Scheduler: Loaded 0 events

2016-10-24 12:29:06 29226 [Note]

/usr/local/mysql/bin/mysqld: ready for connections.

Version: '5.6.19-upsql-0.1.1-log'socket: '/etc/mysock/it_mgmad.sock'port: 60002UpSQL Server

2016-10-24 12:30:21 29226 [ERROR]

/usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)

2016-10-24 12:30:21 29226 [ERROR]

Out of memory; check if mysqld or some other process uses all available memory;

if not, you may have to use 'ulimit' to allow mysqld to use more memory or you

can add more swap space

161024 12:31:35 mysqld_safe Number of

processes running now: 0

161024 12:31:35 mysqld_safe mysqld restarted

2016-10-24 12:31:37 29659

[Warning] Buffered warning: Could not increase number of max_open_files to more

than 10240 (request: 25000)

从上面的信息，我们看到， MSYQL报内存不足的错误无法启动。

2.操作系统日志提示错误

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482]

Mem-Info: