apache + 2 tomcat搭建的集群,在320并发的时候,apache自动宕机,错误信息如下:
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:40:59 2011] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2
[Tue Nov 15 10:41:00 2011] [alert] Child 26597 returned a Fatal error... Apache is exiting!
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Tue Nov 15 10:41:00 2011] [emerg] (43)Identifier removed: couldn't grab the accept mutex
根本原因是系统资源限制造成的,/etc/security/limits.conf配置如下:
* soft nofile 40960
* hard nofile 40960
打开文件数改大了,但是线程数没有改,RHEL6.1中nproc默认是1000个,也需要改大
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 25924
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 40960
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
最后修改limits.conf为
* soft nofile 40960
* hard nofile 40960
* soft nproc 5000
* hard nproc 5000
* soft memlock unlimited
* hard memlock unlimited
apache出现各种错误是,可以参考下面的文章,IBM Http Server的说明文档很细致啊
http://blog.sina.com.cn/s/blog_7dfa89350100wh2f.html