LINUX AND THE MAXIMUM NUMBER OF PROCESSES (THREADS)

文章出处：http://kavassalis.com/2011/03/linux-and-the-maximum-number-of-processes-threads/

    So last night we were debugging some odd Apache behavior when we discovered that RHEL6 has modified the default number of processes( which includes threads in Linux, but I'll refer to it as procs for the purpose of this article ). In RHEL5 ( and prior ), each user is given the default fork init number of threads( which we'll discuss shortly ). It's a huge number usually, 106,496 threads on my RHEL5 dev box here. RHEL6 on the other hand, drops a file into /etc/security/limits.d that limits all users to 1024 procs.

    Now, /etc/security/limits.conf and /etc/security/limits.d are read by PAM's pam_limits.so, so only things that use PAM will ever touch these. Apache starting on a server reboot is not affected by these limits, so Apache will have access to its full 106,496 threads. But if root restarts apache from a shell (/etc/init.d/httpd restart or service httpd restart), Apache will inherit root's proc limit of 1024! This can be verified with a <?php passthru ('ulimit -u');?> via mod_php.

    1024 threads is totally not enough for reasonable operation in Apache. It will begin to kill children off as they seteid() to its designated user as the call fails with errno 11. (seteid() has not always performed an nproc check, this was added in some 2.6 kernel to prevent people from sneaking over the limit) You can work around this by dropping a ulimit -u <much higher number> into /etc/sysconfig/httpd, but my preferred solution is to remove (blank) RHEL6's new /etc/security/limits.d/90-nproc.conf. ( Or I guess you could just never restart your apache, best to reboot the system if theres a problem :P ) Thanks Redhat.

    Back to that large 106,496 number. Thats the maximum number of procs that nick as a user can spawn, even using ulimit -u he can't go higher than that. The system's maximum number of threads ( visible in /proc/sys/kernel/threads-max ) is 212992 ( exactly double ) is decided at boot time by the kernel. I decided to do some digging in the kernel source and the maximum number of threads is calculated thusly:

max_threads = totalram_pages / (8 * THREAD_SIZE / PAGE_SIZE);

( defined in kernel/fork.c, called by init/main.c )

PAGE_SIZE is architecture specific, for x86 it's 4kb ( and 4MB but thats a discussion for another day ), and is calculated via:

#define __AC(X,Y)       (X##Y)
#define _AC(X,Y)        __AC(X,Y)
#define PAGE_SHIFT      12
#define PAGE_SIZE       (_AC(1,UL) << PAGE_SHIFT)

PAGE_SHIFT is a constant for x86, other architectures support alternate (larger) page sizes. THREAD_SIZE is also architecture specific:

#define THREAD_ORDER    1
#define THREAD_SIZE  (PAGE_SIZE << THREAD_ORDER)

THREAD_ORDER is 1 in x86, but can vary depending on the arch. These result in a PAGE_SIZE of 4096 and a THREAD_SIZE 8192 for x86. Back to the formula above, we can fill it in as:

max_threads = totalram_pages / (8 * 8192 / 4096);

becomes

max_threads = totalram_pages / 16 ;

Calculating totalram_pages is a bit tricky since its not just the full system ram allocated to pages. The easiest way to calculate it from /proc/zoneinfo, by summing the spanned pages:

cat /proc/zoneinfo | grep spanned | awk‘{totalpages=totalpages+$2} END {print totalpages}’

3407872

So on my server with 12 gigabytes of memory we've got:

max_threads =  3407872 / 16;

becomes 

max_threads = 212992;

    Now the default maximum number of threads is designed to be able to only consume half the memory from threads alone. But one user should also not be able to consume all of the threads on the system. Back in kernel/fork.c we have:

init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

    So there you have it, the ulimit -u for nick (or root) is now set to 106496 procs(threads). nick is unable to take up all of the processes on a box, even using ulimit, he cannot raise past max_threads/2 (106496, though root can). Unless modified via /proc/sys/kernel/threas-max, a maxed out system will not use more than 50% of the memory to store thread structures. Hopefully you too now know a little bit more about linux process limits! Big thanks to colleague and friend Jim Hull for his troubleshooting and major help last night.

    The exact formulas may vary kernel to kernel, my finding were through the kernel 2.6.37 source and verified on RHEL5's heavily modified 2.6.18-194kernel.