hive 写 es 报EsHadoopRemoteException: index_closed_exception: closed null

最新推荐文章于 2024-04-11 11:27:59 发布

守猫de人

最新推荐文章于 2024-04-11 11:27:59 发布

阅读量791

点赞数

分类专栏： Elasticsearch 文章标签： es6/es7 大数据

本文链接：https://blog.csdn.net/ZMC921/article/details/109716592

版权

Elasticsearch 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

错误由来，es添加xpack后，hive写es总报 index关闭。通过测试，其实数据是可以写入index，且数据量对不上，怀疑是最后提交的时候无法成功，一直提交刷新，最后报错。

Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1605086559607_0014_1_00, diagnostics=[Task failed, taskId=task_1605086559607_0014_1_00_000035, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:370)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:164)
	... 14 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: index_closed_exception: closed
null
	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)
	at org.elasticsearch.hadoop.rest.RestClient.refresh(RestClient.java:275)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.close(BulkProcessor.java:550)
	at org.elasticsearch.hadoop.rest.RestRepository.close(RestRepository.java:219)
	at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.doClose(EsOutputFormat.java:216)
	at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.close(EsHiveOutputFormat.java:75)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:199)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1027)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:620)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:634)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:634)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:634)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:634)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:346)
	... 15 more

解决方案：待更新