mysql restart fail_2019-01-31 MySQL 的一次启动失败的问题定位

MySQL 的一次启动失败的问题定位

问题现象

复现步骤如下：

1)执行重启mysql服务的命令： service mysql restart

2)几秒钟后，出现提示：

$ sudo service mysql start

[sudo] password for lintong:

Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.

3)根据提示：输入systemctl status mysql.service,得到如下的提示信息：

$ systemctl status mysql.service

● mysql.service - MySQL Community Server

Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)

Active: activating (start-post) (Result: exit-code) since 四 2019-01-31 16:22:35 CST; 23s ago

Process: 28233 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)

Process: 28217 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)

Main PID: 28233 (code=exited, status=1/FAILURE); : 28234 (mysql-systemd-s)

Tasks: 2

Memory: 3.7M

CPU: 344ms

CGroup: /system.slice/mysql.service

└─control

├─28234 /bin/bash /usr/share/mysql/mysql-systemd-start post

└─29680 sleep 1

到这里还并不能定位到问题。

4)继续按照提示，输入命令journalctl -xe，输出如下的信息：

$ sudo journalctl -xe

1月 31 16:23:02 Z840 kernel: keepalived[29923]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:05 Z840 systemd[1]: Failed to start MySQL Community Server.

-- Subject: Unit mysql.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit mysql.service has failed.

--

-- The result is failed.

1月 31 16:23:05 Z840 systemd[1]: mysql.service: Unit entered failed state.

1月 31 16:23:05 Z840 systemd[1]: mysql.service: Failed with result 'exit-code'.

1月 31 16:23:05 Z840 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.

1月 31 16:23:05 Z840 systemd[1]: Stopped MySQL Community Server.

-- Subject: Unit mysql.service has finished shutting down

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit mysql.service has finished shutting down.

1月 31 16:23:05 Z840 systemd[1]: Starting MySQL Community Server...

-- Subject: Unit mysql.service has begun start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit mysql.service has begun starting up.

1月 31 16:23:05 Z840 audit[30151]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/30151/status" pid=30151 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=126 ouid=126

1月 31 16:23:05 Z840 audit[30151]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=30151 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=126 ouid=0

1月 31 16:23:05 Z840 audit[30151]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/30151/status" pid=30151 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=126 ouid=126

1月 31 16:23:06 Z840 kernel: show_signal_msg: 29 callbacks suppressed

1月 31 16:23:06 Z840 kernel: keepalived[30201]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:06 Z840 kernel: keepalived[30213]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:06 Z840 kernel: keepalived[30218]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:06 Z840 kernel: keepalived[30230]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:06 Z840 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE

1月 31 16:23:06 Z840 kernel: keepalived[30238]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:06 Z840 kernel: keepalived[30245]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:07 Z840 kernel: keepalived[30256]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:07 Z840 kernel: keepalived[30261]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:07 Z840 kernel: keepalived[30274]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:07 Z840 kernel: keepalived[30280]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:11 Z840 kernel: show_signal_msg: 27 callbacks suppressed

1月 31 16:23:11 Z840 kernel: keepalived[30521]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:11 Z840 kernel: keepalived[30534]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:11 Z840 kernel: keepalived[30538]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:11 Z840 kernel: keepalived[30546]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:11 Z840 kernel: keepalived[30553]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:12 Z840 kernel: keepalived[30565]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:12 Z840 kernel: keepalived[30578]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:12 Z840 kernel: keepalived[30582]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:12 Z840 kernel: keepalived[30591]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:12 Z840 kernel: keepalived[30597]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:16 Z840 kernel: show_signal_msg: 25 callbacks suppressed

1月 31 16:23:16 Z840 kernel: keepalived[30833]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:16 Z840 kernel: keepalived[30846]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]

1月 31 16:23:16 Z840 sudo[30850]: lintong : TTY=pts/20 ; PWD=/home/lintong ; USER=root ; COMMAND=/bin/journalctl -xe

1月 31 16:23:16 Z840 sudo[30850]: pam_unix(sudo:session): session opened for user root by lintong(uid=0)

5)MySQL的运行日志，在/var/log/mysql/下能找到，但是发现大小为空。

# lintong @ Z840 in / [17:30:27]

$ sudo ls -al /var/log/mysql/

[sudo] password for lintong:

total 64

drwxr-x--- 2 mysql adm 4096 1月 31 16:17 .

drwxrwxr-x 20 root syslog 4096 1月 31 16:31 ..

-rw-r----- 1 mysql adm 0 1月 31 17:30 error.log

-rw-r----- 1 mysql adm 20 1月 30 07:35 error.log.1.gz

-rw-r----- 1 mysql adm 986 1月 29 15:40 error.log.2.gz

-rw-r----- 1 mysql adm 618 1月 29 00:31 error.log.3.gz

-rw-r----- 1 mysql adm 20 1月 27 07:35 error.log.4.gz

-rw-r----- 1 mysql adm 20 1月 26 07:35 error.log.5.gz

-rw-r----- 1 mysql adm 2258 1月 25 17:47 error.log.6.gz

-rw-r----- 1 mysql adm 20 1月 25 06:40 error.log.7.gz

-rw-r----- 1 mysql adm 0 1月 31 07:35 general.log

-rw-r----- 1 mysql adm 20 1月 30 07:35 general.log.1.gz

-rw-r----- 1 mysql adm 20 1月 29 07:35 general.log.2.gz

-rw-r----- 1 mysql adm 20 1月 28 07:35 general.log.3.gz

-rw-r----- 1 mysql adm 20 1月 27 07:35 general.log.4.gz

-rw-r----- 1 mysql adm 20 1月 26 07:35 general.log.5.gz

-rw-r----- 1 mysql adm 20 1月 25 07:35 general.log.6.gz

-rw-r----- 1 mysql adm 20 1月 24 07:35 general.log.7.gz

问题定位

采用技术问题的七步解决法，优先到Stack Overflow中搜索，发现有一个与我遇到的问题非常相似：

但是在journalctl -xe 日志存在差异，因为事先已经知道目前的机器内存不足，因此怀疑是内存不足引起的。

正好在后面的回答中，也有类似的出现的情况：

问题根因：空间不足。

问题解决

清理磁盘的空间。