hadoop运行随记5

Exception in thread "communication thread" Exception in thread "Thread for syncLogs" Exception in thread "Timer for 'ReduceTask' metrics system" Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: getNewTla
	at java.lang.Throwable.toString(Throwable.java:344)
	at java.lang.String.valueOf(String.java:2826)
	at java.io.PrintWriter.println(PrintWriter.java:710)
	at java.lang.Throwable.printStackTrace(Throwable.java:509)
	at org.apache.hadoop.util.StringUtils.stringifyException(StringUtils.java:60)
	at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:690)
	at java.lang.Thread.run(Thread.java:662)
java.lang.OutOfMemoryError: getNewTla
	at org.apache.log4j.Hierarchy.getCurrentLoggers(Hierarchy.java:314)
	at org.apache.log4j.LogManager.getCurrentLoggers(LogManager.java:232)
	at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:350)
	at org.apache.hadoop.mapred.Child$3.run(Child.java:141)
java.lang.OutOfMemoryError: getNewTla
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotMetrics(MetricsSystemImpl.java:319)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.onTimerEvent(MetricsSystemImpl.java:309)
	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4.run(MetricsSystemImpl.java:296)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
java.lang.OutOfMemoryError

Exception in thread "IPC Client (47) connection to master/192.168.2.20:8020 from maintain" java.lang.OutOfMemoryError
Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError
Exception in thread "Timer for 'ReduceTask' metrics system" java.lang.OutOfMemoryError
Exception in thread "ganglia" java.lang.OutOfMemoryError
Exception in thread "ResponseProcessor for block blk_-1396931942055176863_4723681" java.lang.OutOfMemoryError: getNewTla
	at java.util.HashMap.addEntry(HashMap.java:937)
	at java.util.HashMap.put(HashMap.java:477)
	at java.util.HashSet.add(HashSet.java:200)
	at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:72)
	at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:56)
	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:357)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.DataInputStream.readFully(DataInputStream.java:178)
	at java.io.DataInputStream.readLong(DataInputStream.java:399)
	at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2967)

解决方案备案：

Error java.lang.OutOfMemoryError: getNewTla using Oracle EPM products

By Marc Schumacher on Oct 31, 2012

Running into a Java out of memory error, it is very common behaviour in the field that the Java heap size will be increased. While thismight help to solve a heap space out of memory error, it might not help to fixan out of memory error for the Thread Local Area (TLA). Increasing theavailable heap space from 1 GB to 16 GB might not even help in this situation.

The Thread Local Area (TLA) is part of the Java heap, but asthe name already indicates, this memory area is local to a specific thread sothere is no need to synchronize with other threads using this memory area.

For optimization purposes the TLA size is configurable using the Java command line option “-XXtlasize”. Depending on the JRockit version and the available Java heap, the default values vary.

Using Oracle EPM System (mainly 11.1.2.x) the following setting was tested successfully:

-XXtlasize:min=8k,preferred=128k

More information about the “-XXtlasize” parameter can befound in the JRockit documentation:

http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionXX.html

随着集群数据越来越多，发现使用jrockit的默认参数将不能保证集群稳定、可靠。后期将进一步开始调优jrockit的参数。