Apache起不来，报No space left on device: Couldn't create accept lock

Apache起不来，报No space left on device: Couldn't create accept lock

今天早上，罗明在搭建测试环境的时候，发现启动MA应用后，访问总是出错。报错特别快，怀疑是apache没有起来，telnet apache端口，发现果然没有起来。

forrest@ubuntu:~$ telnet 10.20.142.12 2080
Trying 10.20.142.12...
telnet: Unable to connect to remote host: Connection refused

但是启动脚本显示是已经成功起来，改成访问jboss端口就可以了。

forrest@ubuntu:~$ telnet 10.20.142.12 2087
Trying 10.20.142.12...
Connected to 10.20.142.12.
Escape character is '^]'.

但是问题总是要解决。我以前遇到类似的问题，就是访问jboss端口是可以的，但是访问apache端口则总是报错，但是是报404的错误，说明apache已经起来，但是是mod_jk出错了。具体可以参见笔者以前写的一篇文章：后翌虚拟机环境搭建日志（简言之，就是添加JkMountCopy All配置项）。

一边单独启动apache，一边查看log日志：

scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/bin]./apachectl restart
./apachectl restart: httpd not running, trying to start

scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]tailf error_log 
[Fri Oct 08 18:41:35 2010] [warn] pid file /home/scrm/work/intl-myalibaba/deploy/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Oct 08 18:41:35 2010] [emerg] (28)No space left on device: Couldn't create accept lock
[Fri Oct 08 18:42:44 2010] [warn] pid file /home/scrm/work/intl-myalibaba/deploy/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Oct 08 18:42:44 2010] [emerg] (28)No space left on device: Couldn't create accept lock

发现报了这么一个奇怪的错误：

[emerg] (28)No space left on device: Couldn't create accept lock。

以为是空间不足，造成无法创建锁文件，但是df -h察看，发觉不是这个问题。

scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda3            4.9G  178M  4.4G   4% /
/dev/xvda1             99M   11M   83M  12% /boot
none                  2.6G     0  2.6G   0% /dev/shm
/dev/xvda8             68G   48G   17G  74% /home
/dev/xvda5            2.9G   42M  2.7G   2% /tmp
/dev/xvda2            9.7G  6.1G  3.1G  67% /usr
/dev/xvda6            2.9G  166M  2.6G   6% /var
10.20.131.137:/home/efs/www
                       34G   23G  8.8G  73% /mnt/efs/www
10.20.134.136:/home/efs/www/escrow
                       34G  3.2G   29G  10% /mnt/escrow

google了一下，发现早已经有人遇到同样的问题了。

scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]ipcs -s | grep scrm    
0x00000000 1212419    scrm      600        1         
0x00000000 1245188    scrm      600        1         
0x00000000 1277957    scrm      600        1         
0x00000000 18579474   scrm      600        1         
0x00000000 18612243   scrm      600        1         
0x00000000 18645012   scrm      600        1         
0x00000000 53444632   scrm      600        1         
0x00000000 53477401   scrm      600        1         
0x00000000 53510170   scrm      600        1         
0x00000000 19890215   scrm      600        1         
0x00000000 19922984   scrm      600        1         
0x00000000 19955753   scrm      600        1         
0x00000000 21692458   scrm      600        1         
0x00000000 21725227   scrm      600        1         
0x00000000 21757996   scrm      600        1         
0x00000000 22347824   scrm      600        1         
0x00000000 22380593   scrm      600        1         
0x00000000 22413362   scrm      600        1         
0x00000000 25296950   scrm      600        1         
0x00000000 25329719   scrm      600        1         
0x00000000 25362488   scrm      600        1         
0x00000000 38895685   scrm      600        1         
0x00000000 38928454   scrm      600        1         
0x00000000 38961223   scrm      600        1         
0x00000000 36438091   scrm      600        1         
0x00000000 36470860   scrm      600        1         
0x00000000 36503629   scrm      600        1         
0x00000000 29720660   scrm      600        1         
0x00000000 29753429   scrm      600        1         
0x00000000 29786198   scrm      600        1         
0x00000000 29884503   scrm      600        1         
0x00000000 29917272   scrm      600        1         
0x00000000 29950041   scrm      600        1         
0x00000000 30539866   scrm      600        1         
0x00000000 30572635   scrm      600        1         
0x00000000 30605404   scrm      600        1         
0x00000000 39059564   scrm      600        1         
0x00000000 39092333   scrm      600        1         
0x00000000 39125102   scrm      600        1         
0x00000000 40697970   scrm      600        1         
0x00000000 40730739   scrm      600        1         
0x00000000 40763508   scrm      600        1

察看发觉有一大堆的ipc使用，需要干掉。

不加-s参数，将打印所有IPC信息。 scrm@intlqa142012x:[/home/scrm/work/intl-myalibaba/deploy/logs]ipcs ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status ------ Semaphore Arrays -------- key semid owner perms nsems ------ Message Queues -------- key msqid owner perms used-bytes messages

清除命令:

scrm@intlqa142012x:[/home/scrm]for semid in `ipcs -s | grep scrm | cut -f2 -d" "`;
> do ipcrm -s $semid; done

再次grep一下，发现已经没有了。

scrm@intlqa142012x:[/home/scrm]ipcs -s | grep scrm

然后启动Apache即可。

查看the number of semaphores that are available on the system admin@intlqa142012x:[/home/admin]ipcs -l ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 32768 max total shared memory (kbytes) = 8388608 min seg size (bytes) = 1 ------ Semaphore Limits -------- max number of arrays = 128 max semaphores per array = 250 max semaphores system wide = 32000 max ops per semop call = 32 semaphore max value = 32767 ------ Messages: Limits -------- max queues system wide = 16 max size of message (bytes) = 8192 default max size of queue (bytes) = 16384 如果有权限的话，可以修改这些限制： To change these parameters, modify the file /etc/sysctl.conf and add the following lines: kernel.msgmni = 1024 kernel.sem = 250 256000 32 1024 Then load these settings with the command：sysctl -p -p Load in sysctl settings from the file specified or /etc/sysctl.conf if none given. admin@intlqa142012x:[/home/admin]sysctl -p error: permission denied on key 'net.ipv4.ip_forward' error: permission denied on key 'net.ipv4.conf.default.rp_filter' error: permission denied on key 'net.ipv4.conf.default.accept_source_route' error: permission denied on key 'kernel.sysrq' error: permission denied on key 'kernel.core_uses_pid' Your Apache process should now be able to create the needed semaphores and run properly. 说明：kernel.msgmni 该文件指定消息队列标识的最大数目，即系统范围内最大多少个消息队列。缺省设置：16。 admin@intlqa142012x:[/home/admin]cat /proc/sys/kernel/msgmni 16 在root下可用sysctl kernel.msgmni检查该参数： admin@intlqa142012x:[/home/admin]sysctl kernel.msgmni kernel.msgmni = 16 也可以在命令行下，使用sysctl -w kernel.msgmni=XXX重新设定。 admin@intlqa142012x:[/home/admin]sysctl -w kernel.msgmni=32 error: permission denied on key 'kernel.msgmni'