mysql docker oom_记一次docker mysql被oom排查

最近博客老是出现无法连接数据库故障！

排查腾讯云监控的时候看到内存使用700多M,问题就出在这里，当时没有使用top命令查看，

腾讯云监控的内存使用的计算方式是取 /proc/meminfo 的

Memtotal-MemFree-Buffers-Cached-SReclaimable(SReclaimable代表可回收的 slab 内存大小)

所有与top命令的出入。本服务器没有开启交换空间sawp

所以是因为内存不足导致的oom 通过系统的计算算出docker mysql 的内存占用最大所以直接oom了！

排查出现oom的过程可以查看/var/log/messages 日志情况

实际上是我的php-fpm进程占用内存过多导致的，优化fpf-fpm.conf参数即可，优化需要根据自己的业务场景优化！

就不贴出我的配置了！有需要的可以找我！

日志如下

Dec 26 10:46:02 docker_web systemd: Started Session 370294 of user root.

Dec 26 10:46:22 docker_web kernel: daotunnel invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

Dec 26 10:46:22 docker_web kernel: daotunnel cpuset=/ mems_allowed=0

Dec 26 10:46:22 docker_web kernel: CPU: 0 PID: 30670 Comm: daotunnel Tainted: G ------------ T 3.10.0-957.10.1.el7.x86_64 #1

Dec 26 10:46:22 docker_web kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014

Dec 26 10:46:22 docker_web kernel: Call Trace:

Dec 26 10:46:22 docker_web kernel: [] dump_stack+0x19/0x1b

Dec 26 10:46:22 docker_web kernel: [] dump_header+0x90/0x229

Dec 26 10:46:22 docker_web kernel: [] ? ktime_get_ts64+0x52/0xf0

Dec 26 10:46:22 docker_web kernel: [] ? delayacct_end+0x8f/0xb0

Dec 26 10:46:22 docker_web kernel: [] ? virtballoon_oom_notify+0x2a/0x70 [virtio_balloon]

Dec 26 10:46:23 docker_web kernel: [] oom_kill_process+0x254/0x3d0

Dec 26 10:46:23 docker_web kernel: [] ? oom_unkillable_task+0xcd/0x120

Dec 26 10:46:23 docker_web kernel: [] ? find_lock_task_mm+0x56/0xc0

Dec 26 10:46:23 docker_web kernel: [] out_of_memory+0x4b6/0x4f0

Dec 26 10:46:23 docker_web kernel: [] __alloc_pages_slowpath+0x5d6/0x724

Dec 26 10:46:23 docker_web kernel: [] __alloc_pages_nodemask+0x404/0x420

Dec 26 10:46:23 docker_web kernel: [] alloc_pages_current+0x98/0x110

Dec 26 10:46:23 docker_web kernel: [] __page_cache_alloc+0x97/0xb0

Dec 26 10:46:23 docker_web kernel: [] filemap_fault+0x298/0x490

Dec 26 10:46:23 docker_web kernel: [] ext4_filemap_fault+0x36/0x50 [ext4]

Dec 26 10:46:23 docker_web kernel: [] __do_fault.isra.59+0x8a/0x100

Dec 26 10:46:23 docker_web kernel: [] do_read_fault.isra.61+0x4c/0x1b0

Dec 26 10:46:23 docker_web kernel: [] handle_pte_fault+0x2f4/0xd10

Dec 26 10:46:23 docker_web kernel: [] ? core_sys_select+0x26d/0x340

Dec 26 10:46:23 docker_web kernel: [] ? try_to_wake_up+0x190/0x390

Dec 26 10:46:23 docker_web kernel: [] handle_mm_fault+0x39d/0x9b0

Dec 26 10:46:23 docker_web kernel: [] __do_page_fault+0x203/0x500

Dec 26 10:46:23 docker_web kernel: [] ? poll_select_copy_remaining+0x113/0x180

Dec 26 10:46:23 docker_web kernel: [] trace_do_page_fault+0x56/0x150

Dec 26 10:46:23 docker_web kernel: [] do_async_page_fault+0x22/0xf0

Dec 26 10:46:23 docker_web kernel: [] async_page_fault+0x28/0x30

Dec 26 10:46:23 docker_web kernel: Mem-Info:

Dec 26 10:46:23 docker_web kernel: active_anon:186895 inactive_anon:4166 isolated_anon:0#012 active_file:926 inactive_file:8714 isolated_file:0#012 unevictable:0 dirty:0 writeback

:0 unstable:0#012 slab_reclaimable:17707 slab_unreclaimable:10069#012 mapped:4819 shmem:4583 pagetables:5739 bounce:0#012 free:12245 free_pcp:0 free_cma:0

Dec 26 10:46:23 docker_web kernel: Node 0 DMA free:4588kB min:708kB low:884kB high:1060kB active_anon:8348kB inactive_anon:100kB active_file:12kB inactive_file:1388kB unevictable:

0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:116kB shmem:140kB slab_reclaimable:752kB slab_unreclaimable:29

6kB kernel_stack:0kB pagetables:92kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7575 all_unreclaimable? yes

Dec 26 10:46:23 docker_web kernel: lowmem_reserve[]: 0 972 972 972

Dec 26 10:46:23 docker_web kernel: Node 0 DMA32 free:44392kB min:44344kB low:55428kB high:66516kB active_anon:739232kB inactive_anon:16564kB active_file:3692kB inactive_file:33468

kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032048kB managed:998980kB mlocked:0kB dirty:0kB writeback:0kB mapped:19160kB shmem:18192kB slab_reclaimable:70076

kB slab_unreclaimable:39980kB kernel_stack:4016kB pagetables:22864kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclai

mable? no

Dec 26 10:46:23 docker_web kernel: lowmem_reserve[]: 0 0 0 0

Dec 26 10:46:23 docker_web kernel: Node 0 DMA: 254kB (UEM) 148kB (UE) 816kB (UE) 332kB (EM) 564kB (UEM) 6128kB (UEM) 0256kB 2512kB (UE) 01024kB 12048kB (M) 0*4096kB = 45

96kB

Dec 26 10:46:24 docker_web kernel: Node 0 DMA32: 7644kB (UE) 4038kB (UEM) 31816kB (UEM) 25232kB (UEM) 16664kB (UEM) 66128kB (UEM) 19256kB (UE) 2512kB (UE) 01024kB 02048k

B 0*4096kB = 44392kB

Dec 26 10:46:24 docker_web kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB

Dec 26 10:46:24 docker_web kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB

Dec 26 10:46:24 docker_web kernel: 14228 total pagecache pages

Dec 26 10:46:24 docker_web kernel: 0 pages in swap cache

Dec 26 10:46:24 docker_web kernel: Swap cache stats: add 0, delete 0, find 0/0

Dec 26 10:46:24 docker_web kernel: Free swap = 0kB

Dec 26 10:46:24 docker_web kernel: Total swap = 0kB

Dec 26 10:46:24 docker_web kernel: 262010 pages RAM

Dec 26 10:46:24 docker_web kernel: 0 pages HighMem/MovableOnly

Dec 26 10:46:24 docker_web kernel: 8288 pages reserved

Dec 26 10:46:24 docker_web kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name

Dec 26 10:46:24 docker_web kernel: [ 1298] 0 1298 35445 174 76 0 0 systemd-journal

Dec 26 10:46:24 docker_web kernel: [ 1318] 0 1318 29157 76 26 0 0 lvmetad

Dec 26 10:46:24 docker_web kernel: [ 1329] 0 1329 11261 307 23 0 -1000 systemd-udevd

Dec 26 10:46:24 docker_web kernel: [ 2047] 0 2047 13880 161 27 0 -1000 auditd

Dec 26 10:46:24 docker_web kernel: [ 2162] 999 2162 153634 1459 63 0 0 polkitd

Dec 26 10:46:24 docker_web kernel: [ 2166] 0 2166 6653 165 18 0 0 systemd-logind

Dec 26 10:46:24 docker_web kernel: [ 2167] 998 2167 2144 38 10 0 0 lsmd

Dec 26 10:46:24 docker_web kernel: [ 2169] 81 2169 15052 167 35 0 -900 dbus-daemon

Dec 26 10:46:24 docker_web kernel: [ 2192] 38 2192 12339 227 30 0 0 ntpd

Dec 26 10:46:24 docker_web kernel: [ 2264] 0 2264 1096 35 7 0 0 acpid

Dec 26 10:46:24 docker_web kernel: [ 2767] 0 2767 6476 53 18 0 0 atd

Dec 26 10:46:24 docker_web kernel: [ 2768] 0 2768 31579 219 17 0 0 crond

Dec 26 10:46:24 docker_web kernel: [ 2819] 0 2819 26866 497 51 0 0 dhclient

Dec 26 10:46:24 docker_web kernel: [ 2895] 0 2895 28216 256 57 0 -1000 sshd

Dec 26 10:46:24 docker_web kernel: [ 2898] 0 2898 144002 2798 96 0 0 tuned

Dec 26 10:46:24 docker_web kernel: [ 2900] 0 2900 74400 3188 30 0 0 daomonit

Dec 26 10:46:24 docker_web kernel: [ 2904] 0 2904 182656 3964 187 0 0 rsyslogd

Dec 26 10:46:24 docker_web kernel: [ 2933] 0 2933 119544 4818 56 0 0 containerd

Dec 26 10:46:24 docker_web kernel: [ 2935] 0 2935 131843 10077 106 0 -500 dockerd

Dec 26 10:46:24 docker_web kernel: [ 3007] 0 3007 27526 32 10 0 0 agetty

Dec 26 10:46:24 docker_web kernel: [ 3008] 0 3008 27526 33 10 0 0 agetty

Dec 26 10:46:24 docker_web kernel: [ 3426] 0 3426 24364 170 18 0 0 sgagent

Dec 26 10:46:24 docker_web kernel: [ 4706] 0 4706 57223 3392 67 0 0 python

Dec 26 10:46:24 docker_web kernel: [ 4711] 0 4711 26998 28 10 0 0 tail

Dec 26 10:46:24 docker_web kernel: [22308] 99 22308 28438 1647 19 0 0 frps

Dec 26 10:46:24 docker_web kernel: [30669] 0 30669 117389 795 30 0 0 daotunnel

Dec 26 10:46:24 docker_web kernel: [ 3871] 0 3871 23820 1001 40 0 0 nginx

Dec 26 10:46:24 docker_web kernel: [ 7638] 995 7638 29672 9686 61 0 0 nginx

Dec 26 10:46:24 docker_web kernel: [11183] 0 11183 114532 1489 113 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11184] 48 11184 138272 4760 128 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11185] 48 11185 138236 4703 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11186] 48 11186 138067 4476 126 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11188] 48 11188 138239 4698 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11189] 48 11189 138235 4703 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11210] 48 11210 138236 4679 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11240] 48 11240 138265 4664 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 1517] 48 1517 138215 4623 129 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 1519] 48 1519 138064 4018 126 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 1520] 48 1520 138258 4574 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 3953] 48 3953 138215 4339 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 5261] 48 5261 138229 4265 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 5282] 48 5282 138571 4223 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 4554] 48 4554 138228 4565 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 6115] 48 6115 138228 4260 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 6121] 48 6121 135800 4579 121 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 6122] 48 6122 138039 4369 125 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 9013] 48 9013 138566 4186 126 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 9016] 48 9016 138231 4250 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 9017] 48 9017 138213 4257 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [11215] 48 11215 138253 4258 127 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [ 9344] 48 9344 138075 4020 126 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [14702] 48 14702 138252 4574 130 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [14707] 48 14707 138044 4042 125 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [14708] 48 14708 135784 4270 119 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [14694] 0 14694 28825 65 13 0 0 sh

Dec 26 10:46:24 docker_web kernel: [14695] 0 14695 5139 97 16 0 0 luajit

Dec 26 10:46:24 docker_web kernel: [14959] 0 14959 28825 65 13 0 0 sh

Dec 26 10:46:24 docker_web kernel: [14960] 0 14960 5139 100 15 0 0 luajit

Dec 26 10:46:24 docker_web kernel: [19278] 0 19278 38810 1645 29 0 0 barad_agent

Dec 26 10:46:24 docker_web kernel: [19285] 0 19285 39548 1966 32 0 0 barad_agent

Dec 26 10:46:24 docker_web kernel: [19286] 0 19286 169263 3062 51 0 0 barad_agent

Dec 26 10:46:24 docker_web kernel: [12748] 0 12748 25598 175 21 0 0 YDLive

Dec 26 10:46:24 docker_web kernel: [30302] 0 30302 28825 64 13 0 0 sh

Dec 26 10:46:24 docker_web kernel: [30303] 0 30303 5139 140 17 0 0 luajit

Dec 26 10:46:24 docker_web kernel: [ 933] 0 933 28825 64 13 0 0 sh

Dec 26 10:46:24 docker_web kernel: [ 936] 0 936 5139 125 16 0 0 luajit

Dec 26 10:46:24 docker_web kernel: [ 4338] 0 4338 28825 64 13 0 0 sh

Dec 26 10:46:24 docker_web kernel: [ 4340] 0 4340 5139 96 17 0 0 luajit

Dec 26 10:46:24 docker_web kernel: [14513] 48 14513 116384 3419 115 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [27451] 48 27451 116126 3168 115 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [27460] 48 27460 116383 3399 115 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [18774] 48 18774 116766 3390 116 0 0 php-fpm

Dec 26 10:46:24 docker_web kernel: [13083] 0 13083 27241 288 17 0 -500 docker-proxy

Dec 26 10:46:24 docker_web kernel: [13091] 0 13091 2680 300 10 0 -999 containerd-shim

Dec 26 10:46:24 docker_web kernel: [13108] 999 13108 240779 27745 132 0 0 mysqld

Dec 26 10:46:24 docker_web kernel: [ 2697] 0 2697 49045 13859 63 0 0 YDService

Dec 26 10:46:24 docker_web kernel: [ 7302] 0 7302 39185 383 79 0 0 sshd

Dec 26 10:46:24 docker_web kernel: [ 7315] 0 7315 29196 517 13 0 0 bash

Dec 26 10:46:24 docker_web kernel: [16556] 0 16556 251102 23921 186 0 0 yum

Dec 26 10:46:24 docker_web kernel: Out of memory: Kill process 13108 (mysqld) score 109 or sacrifice child

Dec 26 10:46:24 docker_web kernel: Killed process 13108 (mysqld) total-vm:963116kB, anon-rss:110980kB, file-rss:0kB, shmem-rss:0kB

Dec 26 10:46:25 docker_web containerd: time="2019-12-26T10:46:25.042577073+08:00" level=info msg="shim reaped" id=08f9f376ee7297c89c5ccf377ee4ffdce64a3b1d04505f37a1399ea3f903e2c3

Dec 26 10:46:25 docker_web dockerd: time="2019-12-26T10:46:25.081242626+08:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*event

s.TaskDelete"

Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state

Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state

Dec 26 10:46:25 docker_web kernel: device veth85673a7 left promiscuous mode

Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state

Dec 26 10:46:36 docker_web yum[16556]: Updated: ntpdate-4.2.6p5-29.el7.centos.x86_64

Dec 26 10:46:37 docker_web yum[16556]: Updated: ntp-4.2.6p5-29.el7.centos.x86_64

Dec 26 10:46:37 docker_web systemd: Reloading.

Dec 26 10:46:37 docker_web systemd: Cannot add dependency job for unit rpcbind.socket, ignoring: Unit not found.

Dec 26 10:46:37 docker_web systemd: Stopping Network Time Service...

Dec 26 10:46:38 docker_web ntpd[2192]: ntpd exiting on signal 15

Dec 26 10:46:38 docker_web systemd: Stopped Network Time Service.

Dec 26 10:46:38 docker_web systemd: Starting Network Time Service...

Dec 26 10:46:38 docker_web ntpd[16820]: ntpd 4.2.6p5@1.2349-o Thu Aug 8 11:47:59 UTC 2019 (1)

Dec 26 10:46:38 docker_web systemd: Started Network Time Service.

Dec 26 10:46:38 docker_web ntpd[16821]: proto: precision = 0.050 usec

Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c01d 0d kern kernel time sync enabled

Dec 26 10:46:38 docker_web ntpd[16821]: Listen normally on 0 lo 127.0.0.1 UDP 123

Dec 26 10:46:38 docker_web ntpd[16821]: Listen normally on 1 eth0 172.30.0.5 UDP 123

Dec 26 10:46:38 docker_web ntpd[16821]: Listening on routing socket on fd #18 for interface updates

Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c016 06 restart

Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c012 02 freq_set kernel -22.321 PPM

Dec 26 10:46:38 docker_web systemd: Reloading.

Dec 26 10:46:45 docker_web ntpd[16821]: 0.0.0.0 c615 05 clock_sync

Dec 26 10:47:01 docker_web systemd: Started Session 370295 of user root.

Dec 26 10:48:01 docker_web systemd: Started Session 370296 of user root.

Dec 26 10:49:01 docker_web systemd: Started Session 370297 of user root.

Dec 26 10:50:01 docker_web systemd: Started Session 370298 of user root.

Dec 26 10:50:01 docker_web systemd: Started Session 370299 of user root.

Dec 26 10:51:01 docker_web systemd: Started Session 370300 of user root.

Dec 26 10:51:04 docker_web systemd: Started Session 370301 of user root.

Dec 26 10:51:04 docker_web systemd-logind: New session 370301 of user root.

Dec 26 10:51:04 docker_web dbus[2169]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)

Dec 26 10:51:04 docker_web dbus[2169]: [system] Successfully activated service 'org.freedesktop.problems'

Dec 26 10:52:01 docker_web systemd: Started Session 370302 of user root.

Dec 26 10:53:01 docker_web systemd: Started Session 370303 of user root.

Dec 26 10:54:01 docker_web systemd: Started Session 370304 of user root.

Dec 26 10:54:40 docker_web systemd: Started Session 370305 of user root.

Dec 26 10:54:40 docker_web systemd-logind: New session 370305 of user root.

Dec 26 10:54:40 docker_web dbus[2169]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)

Dec 26 10:54:40 docker_web dbus[2169]: [system] Successfully activated service 'org.freedesktop.problems'

Dec 26 10:55:01 docker_web systemd: Started Session 370306 of user root.

Dec 26 10:56:01 docker_web systemd: Started Session 370307 of user root.

Dec 26 10:56:23 docker_web systemd-logind: Removed session 370305.