最近我们测试环境tomcat服务频繁跪了,通过系统日志(/var/log/messages)发现触发了linux OOM killer,tomcat进程被kill掉了.于是去了解了下linux OOM killer机制.
Linux 内核有个机制叫OOM killer(Out Of Memory killer)该机制会监控那些占用内存过大,尤其是瞬间占用内存很快的进程,然后防止内存耗尽而自动把该进程杀掉.内核检测到系统内存不足、挑选并杀掉某个进程的过程可以参考内核源代码linux/mm/oom_kill.c,当系统内存不足的时候,out_of_memory()被触发,然后调用select_bad_process()选择一个"bad"进程杀掉如何判断和选择一个"bad进程呢,linux选择"bad"进程是通过调用oom_badness(),挑选的算法和想法都很简单很朴实:最bad的那个进程就是那个最占用内存的进程。
/**
* oom_badness - heuristic function to determine which candidate task to kill
* @p: task struct of which task we should calculate
* @totalpages: total present RAM allowed for page allocation
*
* The heuristic for determining which task to kill is made to be as simple and
* predictable as possible. The goal is to return the highest value for the
* task consuming the most memory to avoid subsequent oom failures.
*/
unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg,
const nodemask_t *nodemask, unsigned long totalpages)
{
long points;
long adj;
if (oom_unkillable_task(p, memcg, nodemask))
return 0;
p = find_lock_task_mm(p);
if (!p)
return 0;
adj = (long)p->signal->oom_score_adj;
if (adj == OOM_SCORE_ADJ_MIN) {
task_unlock(p);
return 0;
}
/*
* The baseline for the badness score is the proportion of RAM that each
* task's rss, pagetable and swap space use.
*/
points = get_mm_rss(p->mm) + p->mm->nr_ptes +
get_mm_counter(p->mm, MM_SWAPENTS);
task_unlock(p);
/*
* Root processes get 3% bonus, just like the __vm_enough_memory()
* implementation used by LSMs.
*/
if (has_capability_noaudit(p, CAP_SYS_ADMIN))
adj -= 30;
/* Normalize to oom_score_adj units */
adj *= totalpages / 1000;
points += adj;
/*
* Never return 0 for an eligible task regardless of the root bonus and
* oom_score_adj (oom_score_adj can't be OOM_SCORE_ADJ_MIN here).
*/
return points > 0 ? points : 1;
}
明白了oom killer机制,就知道怎么去解决防止tomcat服务被kill了,最简单的办法就是增加内存,还有一个临时的办法就是调整内核参数,让 tomcat 进程不容易被 OOM killer 发现,设置参数/pro/pid/oom_adj为 -17,可禁止被系统内核杀掉。 内核会通过特定的算法给每个进程计算一个分数来决定杀哪个进程,每个进程的oom分数以/proc/pid/oos_score中找到。
至于为什么用-17而不用其他数值(默认值为0),这个是由linux内核定义的,查看内核源码可知:以linux-3.3.6版本的kernel源码为例,路径为linux-3.6.6/include/linux/oom.h,阅读内核源码可知oom_adj的可调值为15到-16,其中15最大-16最小,-17为禁止使用OOM.oom_score为2的n次方计算出来的,其中n就是进程的oom_adj值,所以oom_score的分数越高就越会被内核优先杀掉。