linux错误日志为.out,linux主机出现out of memory错误

Mar 14 18:16:16 hmsdb1 kernel: Mem-info:

Mar 14 18:16:16 hmsdb1 kernel: Zone

MA freepages:  2696 min:     0 low:     0 high:     0

Mar 14 18:16:16 hmsdb1 kernel: Zone:Normal freepages:  1227 min:  1279 low:  4544 high:  6304

Mar 14 18:16:16 hmsdb1 kernel: Zone:HighMem freepages:139078 min:   255 low: 20992 high: 31488

Mar 14 18:16:16 hmsdb1 kernel: Free pages:      143001 (139078 HighMem)

Mar 14 18:16:16 hmsdb1 kernel: ( Active: 439970/11996, inactive_laundry: 1839, inactive_clean: 1871, free: 143002 )

Mar 14 18:16:16 hmsdb1 kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2696

Mar 14 18:16:16 hmsdb1 kernel:   aa:1 ac:196 id:111 il:2 ic:0 fr:1227

Mar 14 18:16:16 hmsdb1 kernel:   aa:419832 ac:19942 id:11885 il:1837 ic:1871 fr:139078

Mar 14 18:16:16 hmsdb1 kernel: 0*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 10784kB)

Mar 14 18:16:16 hmsdb1 kernel: 221*4kB 53*8kB 29*16kB 10*32kB 6*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4908kB)

Mar 14 18:16:16 hmsdb1 kernel: 2112*4kB 3185*8kB 8005*16kB 3712*32kB 1295*64kB 549*128kB 202*256kB 76*512kB 29*1024kB 1*2048kB 0*409

6kB = 556312kB)

Mar 14 18:16:16 hmsdb1 kernel: Swap cache: add 147175, delete 143291, find 1330942/1338061, race 0+0

Mar 14 18:16:16 hmsdb1 kernel: 188659 pages of slabcache

Mar 14 18:16:16 hmsdb1 kernel: 1302 pages of kernel stacks

Mar 14 18:16:16 hmsdb1 kernel: 0 lowmem pagetables, 219433 highmem pagetables

Mar 14 18:16:16 hmsdb1 kernel: 32 bounce buffer pages, 32 are on the emergency list

Mar 14 18:16:16 hmsdb1 kernel: Free swap:       3905144kB

Mar 14 18:16:17 hmsdb1 kernel: 1572864 pages of RAM

Mar 14 18:16:17 hmsdb1 kernel: 817645 pages of HIGHMEM

Mar 14 18:16:17 hmsdb1 kernel: 556294 reserved pages

Mar 14 18:16:17 hmsdb1 kernel: 71343058 pages shared

Mar 14 18:16:17 hmsdb1 kernel: 3884 pages swap cached

Mar 14 18:16:17 hmsdb1 kernel: Out of Memory: Killed process 27406 (oracle).

Mar 14 18:16:17 hmsdb1 kernel: Fixed up OOM kill of mm-less task

Mar 14 19:01:43 hmsdb1 sshd(pam_unix)[833]: session opened for user root by (uid=0)

Mar 14 19:05:00 hmsdb1 sshd(pam_unix)[833]: session closed for user root

Mar 14 19:06:15 hmsdb1 kernel: Mem-info:

Mar 14 19:06:15 hmsdb1 kernel: Zone

MA freepages:  2696 min:     0 low:     0 high:     0

Mar 14 19:06:15 hmsdb1 kernel: Zone:Normal freepages:   842 min:  1279 low:  4544 high:  6304

Mar 14 19:06:15 hmsdb1 kernel: Zone:HighMem freepages:154781 min:   255 low: 20992 high: 31488

Mar 14 19:06:15 hmsdb1 kernel: Free pages:      158351 (154781 HighMem)

Mar 14 19:06:15 hmsdb1 kernel: ( Active: 439124/11917, inactive_laundry: 1915, inactive_clean: 1890, free: 158351 )

Mar 14 19:06:15 hmsdb1 kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2696

Mar 14 19:06:15 hmsdb1 kernel:   aa:1 ac:459 id:109 il:33 ic:0 fr:874

Mar 14 19:06:15 hmsdb1 kernel:   aa:418603 ac:20061 id:11917 il:1773 ic:1890 fr:154781

Mar 14 19:06:15 hmsdb1 kernel: 0*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 10784kB)

Mar 14 19:06:15 hmsdb1 kernel: 99*4kB 5*8kB 3*16kB 10*32kB 4*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3492kB)

Mar 14 19:06:15 hmsdb1 kernel: 15659*4kB 3927*8kB 8115*16kB 3739*32kB 1294*64kB 550*128kB 202*256kB 76*512kB 29*1024kB 1*2048kB 0*40

96kB = 619124kB)

Mar 14 19:06:15 hmsdb1 kernel: Swap cache: add 147449, delete 143662, find 1332855/1340248, race 0+0

Mar 14 19:06:15 hmsdb1 kernel: 188711 pages of slabcache

Mar 14 19:06:15 hmsdb1 kernel: 1304 pages of kernel stacks

Mar 14 19:06:15 hmsdb1 kernel: 0 lowmem pagetables, 205380 highmem pagetables

Mar 14 19:06:15 hmsdb1 kernel: 32 bounce buffer pages, 32 are on the emergency list

Mar 14 19:06:15 hmsdb1 kernel: Free swap:       3914208kB

Mar 14 19:06:15 hmsdb1 kernel: 1572864 pages of RAM

Mar 14 19:06:15 hmsdb1 kernel: 817645 pages of HIGHMEM

Mar 14 19:06:15 hmsdb1 kernel: 556294 reserved pages

Mar 14 19:06:15 hmsdb1 kernel: 66330875 pages shared

Mar 14 19:06:15 hmsdb1 kernel: 3787 pages swap cached

Mar 14 19:06:15 hmsdb1 kernel: Out of Memory: Killed process 5841 (oracle).

Mar 14 19:06:15 hmsdb1 kernel: 1304 pages of kernel stacks

Mar 14 19:06:15 hmsdb1 kernel: 0 lowmem pagetables, 205380 highmem pagetables

Mar 14 19:06:15 hmsdb1 kernel: 32 bounce buffer pages, 32 are on the emergency list

Mar 14 19:06:15 hmsdb1 kernel: Free swap:       3914208kB

Mar 14 19:06:15 hmsdb1 kernel: 1572864 pages of RAM

Mar 14 19:06:15 hmsdb1 kernel: 817645 pages of HIGHMEM

Mar 14 19:06:15 hmsdb1 kernel: 556294 reserved pages

Mar 14 19:06:15 hmsdb1 kernel: 66330875 pages shared

Mar 14 19:06:15 hmsdb1 kernel: 3787 pages swap cached

Mar 14 19:06:15 hmsdb1 kernel: Out of Memory: Killed process 5841 (oracle).

Mar 14 19:06:15 hmsdb1 kernel: Fixed up OOM kill of mm-less task

Mar 14 19:06:51 hmsdb1 kernel: Mem-info:

Mar 14 19:06:51 hmsdb1 kernel: Zone

MA freepages:  2696 min:     0 low:     0 high:     0

Mar 14 19:06:51 hmsdb1 kernel: Zone:Normal freepages:   775 min:  1279 low:  4544 high:  6304

Mar 14 19:06:51 hmsdb1 kernel: Zone:HighMem freepages:150990 min:   255 low: 20992 high: 31488

Mar 14 19:06:51 hmsdb1 kernel: Free pages:      154470 (150990 HighMem)

Mar 14 19:06:51 hmsdb1 kernel: ( Active: 442870/11919, inactive_laundry: 1901, inactive_clean: 1890, free: 154470 )

Mar 14 19:06:51 hmsdb1 kernel:   aa:0 ac:0 id:0 il:0 ic:0 fr:2696

Mar 14 19:06:51 hmsdb1 kernel:   aa:1 ac:581 id:0 il:128 ic:0 fr:784

Mar 14 19:06:51 hmsdb1 kernel:   aa:422219 ac:20069 id:11919 il:1773 ic:1890 fr:150990

Mar 14 19:06:51 hmsdb1 kernel: 0*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 10784kB)

Mar 14 19:06:51 hmsdb1 kernel: 27*4kB 6*8kB 12*16kB 5*32kB 3*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3132kB)

Mar 14 19:06:51 hmsdb1 kernel: 11852*4kB 3935*8kB 8115*16kB 3739*32kB 1294*64kB 550*128kB 202*256kB 76*512kB 29*1024kB 1*2048kB 0*40

96kB = 603960kB)

Mar 14 19:06:51 hmsdb1 kernel: Swap cache: add 147449, delete 143662, find 1333002/1340395, race 0+0

Mar 14 19:06:51 hmsdb1 kernel: 188676 pages of slabcache

Mar 14 19:06:51 hmsdb1 kernel: 1314 pages of kernel stacks

Mar 14 19:06:51 hmsdb1 kernel: 0 lowmem pagetables, 206211 highmem pagetables

Mar 14 19:06:51 hmsdb1 kernel: 32 bounce buffer pages, 32 are on the emergency list

Mar 14 19:06:51 hmsdb1 kernel: Free swap:       3914208kB

Mar 14 19:06:51 hmsdb1 kernel: 1572864 pages of RAM

Mar 14 19:06:51 hmsdb1 kernel: 817645 pages of HIGHMEM

Mar 14 19:06:51 hmsdb1 kernel: 556294 reserved pages

Mar 14 19:06:51 hmsdb1 kernel: 66363658 pages shared

Mar 14 19:06:51 hmsdb1 kernel: 3787 pages swap cached

Mar 14 19:06:51 hmsdb1 kernel: Out of Memory: Killed process 22109 (oracle).

如上日志所示：

lowmem不断减少

221*4kB 53*8kB 29*16kB 10*32kB 6*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 4908kB

变成

99*4kB 5*8kB 3*16kB 10*32kB 4*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3492kB

但highmem有在增加

2112*4kB 3185*8kB 8005*16kB 3712*32kB 1295*64kB 549*128kB 202*256kB 76*512kB 29*1024kB 1*2048kB 0*409

6kB = 556312kB

变成

15659*4kB 3927*8kB 8115*16kB 3739*32kB 1294*64kB 550*128kB 202*256kB 76*512kB 29*1024kB 1*2048kB 0*40

96kB = 619124kB

此处有点疑问的是2112变成15659究竟是不是碎片增加了。

并且此时通过日志看到，swap应该还有换入换出： Swap cache: add 147449, delete 143662, find 1332855/1340248, race 0+0

不过越到后面，SWAP就只有增加而没有删除了，呵呵，看来还真的是遇到lowmem问题了？如何解决