oom killer理解和日志分析

前言

我们环境上接到报警，我们的应用停止运行了。环境上无dump文件保留下来（配置如果发生oom保存dump 文件），当我们查看系统日志

oom-killer查看各个日志含义

/var/log/messages 发现 系统中有文件调用 oom-killer 这个命令。
由于环境上面的日志文件不能发出来这里发出几个比较关键点用于以后直接定位日志出现

Sep 28 23:50:01 VM-0-9-centos kernel: node invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Sep 28 23:50:01 VM-0-9-centos kernel: node cpuset=/ mems_allowed=0

这里 node invoked oom-killer 根据这个日志可以看出来是node 的创建导致内存不足调用oom-killer

Sep 28 14:50:01 VM-0-9-centos kernel: Out of memory: Kill process 5430 (java) score 800 or sacrifice child
Sep 28 14:50:01 VM-0-9-centos kernel: Killed process 5430 (java) total-vm:124654756kB, anon-rss:2428635kB, file-rss:0kB

在一份oom-killer日志末尾处 Killed process 表明杀死那个进程 5430 进程，5430 占用anon-rss:2428635kB（2.3G）大小。

Sep 28 14:50:01 VM-0-9-centos kernel: Node 0 DMA free:15900kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Sep 28 14:50:01 VM-0-9-centos kernel: lowmem_reserve[]: 0 2801 7792 7792

Sep 28 14:50:01 VM-0-9-centos kernel: Node 0 DMA32 free:43500kB min:24252kB low:30312kB high:36376kB
active_anon:2643608kB(2.5G)  inactive_anon:61560kB active_file:40kB inactive_file:40kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2869240kB mlocked:0kB dirty:0kB writeback:0kB mapped:748kB shmem:160024kB slab_reclaimable:54996kB slab_unreclaimable:6816kB kernel_stack:704kB pagetables:67440kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:275 all_unreclaimable? yes
Sep 28 14:50:01 VM-0-9-centos kernel: lowmem_reserve[]: 0 0 4990 4990

Sep 28 14:50:01 VM-0-9-centos kernel: Node 0 Normal free:35100kB min:43192kB low:53988kB high:64788kB
active_anon:4609420kB(4.3G) inactive_anon:87644kB active_file:296kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5242880kB managed:5110124kB mlocked:0kB dirty:0kB writeback:0kB mapped:4260kB shmem:242100kB slab_reclaimable:81876kB slab_unreclaimable:15720kB kernel_stack:1808kB pagetables:204928kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:511 all_unreclaimable? yes
Sep 28 14:50:01 VM-0-9-centos kernel: lowmem_reserve[]: 0 0 0 0

Node 0 DMA free:15900kB min:132kB
Node 0 DMA32 free:43500kB min:24252kB
Node 0 Normal free:35100kB min:43192kB
分别表明当前存在 Node 0 DMA / Node 0 DMA32/ Node 0 Normal 的空闲内存和最小内存。最小内存是保证系统运行。

Sep 28 14:50:01 VM-0-9-centos kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Sep 28 14:50:01 VM-0-9-centos kernel: [ 5430]  1000  5430 36913689  1537770   66407        0             0 java
Sep 28 14:50:01 VM-0-9-centos kernel: [17132]     0 17132    21804      215      44        0             0 node
Sep 28 14:50:01 VM-0-9-centos kernel: [17133]     0 17133    21804      285      43        0             0 node
Sep 28 14:50:01 VM-0-9-centos kernel: [17134]     0 17134    21866      290      44        0             0 node
Sep 28 14:50:01 VM-0-9-centos kernel: [17135]     0 17135    21866      290      44        0             0 node

这个内存中表明内存中当时运行的的程序（数量很多，这里随便贴出几个用于下次定位）