2017-01-02 15:34:04,461 WARN [IPC Client (188363230) connection to nn01/192.168.1.4:8020 from hive]: ipc.Client (Client.java:run(955)) - Unexpected error reading responses on connection Thread[IPC Client (188363230) connection to nn01.100.cloud/172.16.8.4:8020 from hive,5,main]
java.lang.OutOfMemoryError: Java heap space
at com.google.protobuf.CodedInputStream.<init>(CodedInputStream.java:573)
at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:199)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)
2017-01-02 15:34:03,365 FATAL [Thread-51]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(91)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
at org.apache.thrift.transport.TSocket.<init>(TSocket.java:78)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:114)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:121)
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:86)
at java.lang.Thread.run(Thread.java:745)
2017-01-02 15:34:29,186 INFO [Thread-50]: server.HiveServer2 (HiveServer2.java:stop(266)) - Shutting down HiveServer2
2017-01-02 15:34:29,186 INFO [Thread-50]: thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2017-01-02 15:34:29,188 INFO [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at nn02.100.cloud/192.168.1.5
heapsize太小了,可以适当的调大些
Root cause, the heapsize of hadoop opts is too small, need to increase the size in hive-env.sh as bold:
继续分析查看下进程:
观察2天后发现新生代和老年代都已经爆满,永久代几乎爆满,jvm一直在做full GC,但是没有效果,各个内存区都被占用了。
就日志来看,老年代和持久代的内存只要上去了,就再没有下降过,而当这两块区域饱和后,新生代也出现了这种现象。
java.lang.OutOfMemoryError: Java heap space
at com.google.protobuf.CodedInputStream.<init>(CodedInputStream.java:573)
at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:199)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:949)
2017-01-02 15:34:03,365 FATAL [Thread-51]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(91)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
at org.apache.thrift.transport.TSocket.<init>(TSocket.java:78)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:114)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:121)
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:86)
at java.lang.Thread.run(Thread.java:745)
2017-01-02 15:34:29,186 INFO [Thread-50]: server.HiveServer2 (HiveServer2.java:stop(266)) - Shutting down HiveServer2
2017-01-02 15:34:29,186 INFO [Thread-50]: thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2017-01-02 15:34:29,187 INFO [Thread-50]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2017-01-02 15:34:29,188 INFO [Thread-3]: server.HiveServer2 (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at nn02.100.cloud/192.168.1.5
分析及解决:
java.lang.OutOfMemoryError: Java heap space内存溢出报错
heapsize太小了,可以适当的调大些
Root cause, the heapsize of hadoop opts is too small, need to increase the size in hive-env.sh as bold:
if [ "$SERVICE" = "cli" ]; then
if [ -z "$DEBUG" ]; then
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
else
export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
fi
fi
# The heap size of the jvm stared by hive shell script can be controlled via:
#
export HADOOP_HEAPSIZE=2048
继续分析查看下进程:
监控语句如下:
jstat -gcutil [进程号] 5000 --每5秒打印一次jvm各个内存区的状态
另外找了一个监控系统内存的语句:
vmstat | sed -n '/[0-9]/p'
观察2天后发现新生代和老年代都已经爆满,永久代几乎爆满,jvm一直在做full GC,但是没有效果,各个内存区都被占用了。
就日志来看,老年代和持久代的内存只要上去了,就再没有下降过,而当这两块区域饱和后,新生代也出现了这种现象。
是有什么资源没释放吧?一看代码,果然,有的连接没有关闭!
如果还是有问题,hive0.14版本,自身也存在着bug,可以考虑进行定期重启hiveserver2服务,别笑实际工作中很多大的公司都是这样做的。
参考:
http://blog.csdn.net/gklifg/article/details/50418109
http://blog.csdn.net/lulynn/article/details/46455097