Hive查询时不加limit报错

最新推荐文章于 2022-05-05 19:56:46 发布

一点IT技术

最新推荐文章于 2022-05-05 19:56:46 发布

阅读量1.3k

点赞数 1

分类专栏： Hive 文章标签： hive

本文链接：https://blog.csdn.net/weixin_43239554/article/details/104745065

版权

Hive 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

执行sql语句不加limit时报错：Unknown HS2 problem when communicating with Thrift server.
Error: org.apache.thrift.transport.TTransportException: java.net.SocketException: 断开的管道 (Write failed) (state=08S01,code=0)

查看hive日志

Exception in thread "HiveServer2-Handler-Pool: Thread-38" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:476)
	at org.apache.thrift.transport.TSaslServerTransport.write(TSaslServerTransport.java:41)
	at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:202)
	at org.apache.hive.service.rpc.thrift.TStringColumn$TStringColumnStandardScheme.write(TStringColumn.java:490)
	at org.apache.hive.service.rpc.thrift.TStringColumn$TStringColumnStandardScheme.write(TStringColumn.java:433)
	at org.apache.hive.service.rpc.thrift.TStringColumn.write(TStringColumn.java:371)
	at org.apache.hive.service.rpc.thrift.TColumn.standardSchemeWriteValue(TColumn.java:385)
	at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:244)
	at org.apache.thrift.TUnion$TUnionStandardScheme.write(TUnion.java:213)
	at org.apache.thrift.TUnion.write(TUnion.java:152)
	at org.apache.hive.service.rpc.thrift.TRowSet$TRowSetStandardScheme.write(TRowSet.java:804)
	at org.apache.hive.service.rpc.thrift.TRowSet$TRowSetStandardScheme.write(TRowSet.java:695)
	at org.apache.hive.service.rpc.thrift.TRowSet.write(TRowSet.java:609)
	at org.apache.hive.service.rpc.thrift.TFetchResultsResp$TFetchResultsRespStandardScheme.write(TFetchResultsResp.java:554)
	at org.apache.hive.service.rpc.thrift.TFetchResultsResp$TFetchResultsRespStandardScheme.write(TFetchResultsResp.java:490)
	at org.apache.hive.service.rpc.thrift.TFetchResultsResp.write(TFetchResultsResp.java:416)
	at org.apache.hive.service.rpc.thrift.TCLIService$FetchResults_result$FetchResults_resultStandardScheme.write(TCLIService.java:15916)
	at org.apache.hive.service.rpc.thrift.TCLIService$FetchResults_result$FetchResults_resultStandardScheme.write(TCLIService.java:15880)
	at org.apache.hive.service.rpc.thrift.TCLIService$FetchResults_result.write(TCLIService.java:15831)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:53)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

发现是JVM堆内存溢出
解决方法

在yarn-site.xml中配置

	<property>  
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>4096</value>
	</property>
	<property>
		<name>yarn.scheduler.maximum-allocation-mb</name>
		<value>4096</value>
	</property>
	<property>
		<name>yarn.nodemanager.vmem-pmem-ratio</name>
		<value>2.1</value>
	</property>
	<property>
		<name>mapred.child.java.opts</name>
		<value>-Xmx1024m</value>
	</property>