swapper allocate failed的原因

最近发现线上机器出现一个问题

 

swapper: page allocation failure. order:1, mode:0x20
Pid: 0, comm: swapper Not tainted 2.6.32-71.el6.x86_64 #1

 

dmesg里报出这个信息，然后机器负载开始上升，而实际上用free可以看到还有大量的内存被buffer/cached

因此不应该出这个问题才对，我的系统是centos6，上centos的论坛查了一圈，有人回说是内核的bug

一个临时解决方案是

sysctl -w vm.zone_reclaim_mode=1

关于zone_reclaim_mode的定义kernel的文档里描述如下

Zone_reclaim_mode allows someone to set more or less aggressive approaches to
reclaim memory when a zone runs out of memory. If it is set to zero then no
zone reclaim occurs. Allocations will be satisfied from other zones / nodes
in the system.
 
This is value ORed together of
 
1 = Zone reclaim on
2 = Zone reclaim writes dirty pages out
4 = Zone reclaim swaps pages
 
zone_reclaim_mode is set during bootup to 1 if it is determined that pages
from remote zones will cause a measurable performance reduction. The
page allocator will then reclaim easily reusable pages (those page
cache pages that are currently not used) before allocating off node pages.
 
It may be beneficial to switch off zone reclaim if the system is
used for a file server and all of memory should be used for caching files
from disk. In that case the caching effect is more important than
data locality.
 
Allowing zone reclaim to write out pages stops processes that are
writing large amounts of data from dirtying pages on other nodes. Zone
reclaim will write out dirty pages if a zone fills up and so effectively
throttle the process. This may decrease the performance of a single process

since it cannot use all of system memory to buffer the outgoing writes
anymore but it preserve the memory on other nodes so that the performance
of other processes running on other nodes will not be affected.
 
Allowing regular swap effectively restricts allocations to the local
node unless explicitly overridden by memory policies or cpuset
configurations.

 

总结一下，就是这个参数告诉内核当内存不够用时就直接回收buffer/cache