java.lang.OutOfMemoryError: unable to create new native thread

周末我们在做大并发压力测试时，JBOSS进程会莫名退出.

jia:23

jian hou:22

jia:8

jia:22

jia:23

jian hou:12

jia:21

jian hou:21

jia:8

jia:20

Aug 23, 2014 5:25:54 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException

SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container

java.lang.OutOfMemoryError: unable to create new native thread

        at java.lang.Thread.start0(Native Method)

        at java.lang.Thread.start(Thread.java:640)

        at sun.net.www.http.KeepAliveCache$1.run(KeepAliveCache.java:89)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.net.www.http.KeepAliveCache.put(KeepAliveCache.java:75)

        at sun.net.www.http.HttpClient.putInKeepAliveCache(HttpClient.java:364)

        at sun.net.www.http.HttpClient.finished(HttpClient.java:352)

        at sun.net.www.http.ChunkedInputStream.closeUnderlying(ChunkedInputStream.java:196)

        at sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:428)

        at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:545)

        at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582)

        at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669)

        at java.io.FilterInputStream.read(FilterInputStream.java:116)

        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2672)

        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)

        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)

        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)

        at java.io.InputStreamReader.read(InputStreamReader.java:167)

        at java.io.Reader.read(Reader.java:123)

        at com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:171)

        at com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)

       at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)

        at com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)

        at com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)

        at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:565)

        at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:517)

        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:684)

        at com.sun.jersey.api.client.WebResource.get(WebResource.java:191)

        at com.cup.wm.non.biz.util.RestUtil.getListStr(RestUtil.java:279)

        at com.cup.wm.non.biz.util.RestUtil.getPageInfoFromInterface(RestUtil.java:204)

        at com.cup.wm.non.biz.interfaces.december.BillService.getBillListResult(BillService.java:209)

        at com.cup.wm.non.biz.interfaces.december.BillService.getBillList(BillService.java:194)

        at com.cup.wm.non.biz.web.resource.december.BillResource.getBillList(BillResource.java:135)

        at sun.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)

        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)

        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)^C


7 sys=0.00, real=0.03 secs]

2014-08-23T17:34:44.856+0800: [GC [PSYoungGen: 513778K->4839K(516864K)] 885847K->376940K(1565440K), 0.0241370 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]

2014-08-23T17:34:45.556+0800: [GC [PSYoungGen: 514151K->4324K(516736K)] 886252K->376441K(1565312K), 0.0243180 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]

2014-08-23T17:34:46.274+0800: [GC [PSYoungGen: 513632K->7416K(515328K)] 885749K->380086K(1563904K), 0.0289240 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]

2014-08-23T17:34:46.991+0800: [GC [PSYoungGen: 515320K->3970K(511936K)] 887990K->376703K(1560512K), 0.0232010 secs] [Times: user=0.04 sys=0.01, real=0.02 secs]

2014-08-23T17:34:47.728+0800: [GC [PSYoungGen: 511871K->4263K(516160K)] 884604K->376996K(1564736K), 0.0235360 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]

2014-08-23T17:34:48.436+0800: [GC [PSYoungGen: 512167K->4244K(516032K)] 884900K->377201K(1564608K), 0.0479290 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]

2014-08-23T17:34:49.271+0800: [GC [PSYoungGen: 512148K->5096K(516480K)] 885105K->378213K(1565056K), 0.0283600 secs] [Times: user=0.08 sys=0.01, real=0.02 secs]

2014-08-23T17:34:49.995+0800: [GC [PSYoungGen: 513512K->3592K(516224K)] 886629K->376877K(1564800K), 0.0243280 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]

2014-08-23T17:34:50.745+0800: [GC [PSYoungGen: 511983K->4021K(516608K)] 885268K->377571K(1565184K), 0.0259170 secs] [Times: user=0.05 sys=0.00, real=0.03 secs]

2014-08-23T17:34:51.482+0800: [GC [PSYoungGen: 512821K->4449K(516480K)] 886371K->378071K(1565056K), 0.0267510 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]

2014-08-23T17:34:52.211+0800: [GC [PSYoungGen: 513249K->4016K(516864K)] 886871K->377653K(1565440K), 0.0254580 secs] [Times: user=0.08 sys=0.00, real=0.02 secs]

2014-08-23T17:34:52.942+0800: [GC [PSYoungGen: 513392K->4325K(516800K)] 887029K->377978K(1565376K), 0.0262630 secs] [Times: user=0.08 sys=0.01, real=0.02 secs]

2014-08-23T17:34:53.630+0800: [GC [PSYoungGen: 513701K->3894K(517184K)] 887354K->377571K(1565760K), 0.0265310 secs] [Times: user=0.09 sys=0.00, real=0.03 secs]

2014-08-23T17:34:54.355+0800: [GC [PSYoungGen: 513846K->3886K(517056K)] 887523K->377579K(1565632K), 0.0232410 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]

jia:1

jian hou:0

2014-08-23T17:34:55.059+0800: [GC [PSYoungGen: 513833K->3467K(517312K)] 887527K->377192K(1565888K), 0.0241350 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]

jia:1

jian hou:0

jia:1

jian hou:0

2014-08-23T17:34:55.725+0800: [GC [PSYoungGen: 513739K->3216K(517248K)] 887464K->376957K(1565824K), 0.0279970 secs] [Times: user=0.05 sys=0.01, real=0.03 secs]

jia:1

2014-08-23T17:34:56.559+0800: [GC#

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 2209776 bytes for Chunk::new

# An error report file with more information is saved as:

# /home/yxl/upjas/upjas-minimal/bin/hs_err_pid1414.log

/home/yxl/upjas/upjas-minimal/bin/run.sh: line 286:  1414 Aborted                 "/usr/local/jdk1.6.0_37/bin/java" -Dprogram.name=run.sh -server -XX:+UseParallelOldGC -XX:ParallelGCThreads=2 -XX:PermSize=128m -XX:MaxPermSize=256m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Dorg.jboss.ejb3.remoting.IsLocalInterceptor.passByRef=true -Ddb2.jcc.charsetDecoderEncoder=3 -Dorg.jboss.net.protocol.file.useURI=false -Xms1536m -Xmx1536m -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs="/home/yxl/upjas/upjas-minimal/lib/endorsed" -classpath "/home/yxl/sqllib/java/db2jcc.jar:/home/yxl/sqllib/java/db2jcc_license_cu.jar:/home/yxl/upjas/upjas-minimal/server/default/cup-deploy::/home/yxl/upjas/upjas-minimal/bin/run.jar:/usr/local/jdk1.6.0_37/lib/tools.jar" org.jboss.Main -c default -P instance.properties

In JVM, each thread is allocated separate memory space called"thread stack", which is configured by JVM option "-Xss".The default value depends on OS/JVM. As number of threads increase, memoryusage increases and can lead to 'out ofMemoryError'. Remember, the memory forthreads is allocated on stack and not on Java Heap.

Solution:
If you are getting this error, then you can try decreasing the memory allocatedto heap (-Xmx) and/or decreasing the thread stack memory (-Xss). Increasing-Xmx (32bit systems) will leave less room for threads if it is being used,hence the opposite of the solution.

Remember the memory allocated using '-Xss' is the maximum memoryallocated per thread, and usually it's in KB. Also too low -Xss value can causejava.lang.StackOverflowError. If you continue to get the same error, youprobably have thread leak, fix the program.

创建一个线程时，线程占用的内存不属于堆（heap），而是使用操作系统的内存，也就是说，当物理内存不够时，会导致错误 “java.lang.OutOfMemoryError: unable to create new nativethread”的发生。给jvm设置小一点的-Xss（例如： -Xss128k）可以减少每个线程占用的内存，以便可以创建出更多的线程，不过这只是缓解了该问题。该问题的原因还是因为物理内存不够用了。

补充：

Every thread created in a Java program has its own stack space.The stack space used is not allocated from the heap. Infact if you look at theOS report on the memory used by your JVM, you may notice that it is more thanwhat -Xmx parameter specifies. This is because, beside other things, memory isused for the thread stacks too. And this memory is not included in the heapspecified by the -Xms and -Xmx switches.

The thread stack is used to push stacks frames in nested methodcalls. If the nesting is so deep that the thread runs out of space, the threaddies with a StackOverflowError.

The default thread stack size varies with JVM, OS andenvironment variables. A typical value is 512k. It is generally larger for64bit JVMs because references are 8 bytes rather than 4 bytes in size. Thismeans that if your app uses 150 threads, 75MB will be used for thread stacks.In some environments the defaults stack may be as large as 2MB. With a largenumber of threads, this can consume a significant amount of memory which couldotherwise be used by your application or OS.

In most applications, 128k happens to be enough for the stack.What you really need to do is adjust and observe. If you don’t see your apprunning out of stack space, use the -Xss JVM parameter to specify a smallerstack (-Xss128k).

Note that it is entirely possible that your OS rounds up valuesfor stack size specified by your -Xss parameter. Watch out for that.

Using the program below, you can see how stack space is used upby methods with varying number of arguments. You’ll get the StackOverflowErrorwith fewer nested method calls for a smaller stack. You could try adjusting thenumber of arguments and even the type of arguments for different behaviour.

By adjusting the stack size, and keeping the code the same, youcan see the JVM dying at different points in the recursive call.