glassfish:Resource temporarily unavailable in tsStartJavaThread

         今天测试服务器(用的glassfish3.1.1)在部署应用的时候忽然报出如下异常。随后使用用su和ps都会报出“-bash: fork: retry: Resource temporarily unavailable”。

[ERROR] [ERROR][thread ] Could not start thread Keep-Alive-Timer. Resource temporarily unavailable
[ERROR] Exception in thread "Main Thread" java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1096).
[ERROR] Java heap 3G reserved, 64M committed
[ERROR] Paged memory=18014398505034060K/18389412K.
[ERROR] Your Java heap size might be set too high.
[ERROR] Try to reduce the Java heap size using -Xmx:<size> (e.g. "
[ERROR]         at java.lang.Thread.start0(Native Method)
[ERROR]         at java.lang.Thread.start(Thread.java:597)
[ERROR]         at sun.net.www.http.KeepAliveCache$1.run(KeepAliveCache.java:89)
[ERROR]         at sun.net.www.http.KeepAliveCache.put(KeepAliveCache.java:75)
[ERROR]         at sun.net.www.http.HttpClient.putInKeepAliveCache(HttpClient.java:370)
[ERROR]         at sun.net.www.http.HttpClient.finished(HttpClient.java:358)
[ERROR]         at sun.net.www.http.ChunkedInputStream.closeUnderlying(ChunkedInputStream.java:196)
[ERROR]         at sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:428)
[ERROR]         at sun.net.www.http.ChunkedInputStream.readAheadNonBlocking(ChunkedInputStream.java:493)
[ERROR]         at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:584)
[ERROR]         at sun.net.www.http.ChunkedInputStream.available(ChunkedInputStream.java:698)
[ERROR]         at java.io.FilterInputStream.available(FilterInputStream.java:142)
[ERROR]         at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:196)
[ERROR]         at com.sun.enterprise.util.io.FileUtils.copy(FileUtils.java:915)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteResponseManager.<init>(RemoteResponseManager.java:79)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.handleResponse(RemoteAdminCommand.java:1056)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.handleResponse(RemoteAdminCommand.java:1045)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.access$600(RemoteAdminCommand.java:101)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand$1$1.handleReport(RemoteAdminCommand.java:526)
[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.processReport(PayloadFilesManager.java:473)
[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.access$900(PayloadFilesManager.java:93)
[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager$DataRequestType$4.processPart(PayloadFilesManager.java:776)
[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.processPartsExtended(PayloadFilesManager.java:624)
[ERROR]         at org.glassfish.admin.payload.PayloadFilesManager.processParts(PayloadFilesManager.java:643)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand$1.useConnection(RemoteAdminCommand.java:531)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.doHttpCommand(RemoteAdminCommand.java:700)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.doHttpCommand(RemoteAdminCommand.java:543)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.executeRemoteCommand(RemoteAdminCommand.java:482)
[ERROR]         at com.sun.enterprise.admin.remote.RemoteAdminCommand.executeCommand(RemoteAdminCommand.java:419)
[ERROR]         at com.sun.enterprise.admin.cli.remote.RemoteCommand.executeCommand(RemoteCommand.java:335)
[ERROR]         at com.sun.enterprise.admin.cli.CLICommand.execute(CLICommand.java:264)
[ERROR]         at com.sun.enterprise.admin.cli.AsadminMain.executeCommand(AsadminMain.java:306)
[ERROR]         at com.sun.enterprise.admin.cli.AsadminMain.main(AsadminMain.java:238)
[ERROR] [ERROR][thread ] Could not start thread Thread-0. Resource temporarily unavailable
[ERROR] Deployment of /home/javadev/codes/*****************.war failed.
[ERROR] For more detail on what might be causing the problem try running maven with the --debug option 
[ERROR] or setting the maven-glassfish-plugin "echo" property to "true".

         我们先分析了一下异常，看到“Paged memory=18014398505034060K/18389412K.”立马就吓到了，以为谁无聊又在改配置信息呢。在glassfish控制台查看glassfish启动JVM的配置参数，发现没什么异常，对比了其他配置都没有改变。相同的配置在其他测试服务器都能正常运行，初步排除了glassfish和JVM参数配置的问题。
         随后对“-bash: fork: retry: Resource temporarily unavailable”进行分析，查看进程数:

 [javadev@snaqi-test3 ~]$ ps -ef|wc -l
 1025

         但在/etc/security/limits.conf用户javadev的配置

    javadev             soft    nproc   1024
    javadev             hard    nproc   16384
    javadev             soft    nofile  2048
    javadev             hard    nofile  65536

        显然用户进程数已经达到限制的极限，将第1行的1024改成2048，问题解决。另外，还可以将第3行2048改大，以防程序出现“too many file opens”的异常。

参考：oracle，ulimit，通过 ulimit 改善系统性能