执行Hive报Caused by: java.io.IOException: Unable to rename异常

1、问题描述：

Caused by: java.io.IOException: Unable to rename obs://bigdatahll/warehouse/tablespace/hive/hll_dwb.db/dwb_driver_order_loc_in_dtl/dt=2020-11-09/.hive-staging_hive_2020-11-10_02-36-19_364_6573414509666215386-15624/_task_tmp.-ext-10000/_tmp.000007_0 to obs://bigdatahll/warehouse/tablespace/hive/hll_dwb.db/dwb_driver_order_loc_in_dtl/dt=2020-11-09/.hive-staging_hive_2020-11-10_02-36-19_364_6573414509666215386-15624/_tmp.-ext-10000/000007_0
at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
... 20 more

2、问题解释：

设置tez.am.container.reuse.enabled = true允许容器在多个任务之间重用。 当两个文件合并任务在同一容器上运行时，最后一个任务无法重命名输出路径。

以下是容器container_e87_1570604853053_11564_01_000003上的任务000001_0的错误日志，其中任务000004_0在任务000001_0之前运行。

它显示任务000001_0的输出文件名是错误地从先前的任务ID 000004_0中获取的。

2019-10-15 13:00:31,438 [ERROR] [TezChild] |tez.TezProcessor|: java.lang.RuntimeException: Hive Runtime Error while closing operators
	at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:188)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:284)
	at org.apache.hadoop.hive.ql.exec.tez.MergeFileTezProcessor.run(MergeFileTezProcessor.java:42)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to close AbstractFileMergeOperator
	at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:315)
	at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:265)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:733)
	at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)
	... 17 more
Caused by: java.io.IOException: Unable to rename viewfs://<cluster_name>/user/<user_name>/.hive-staging_hive_2019-10-15_12-59-32_916_2461818728035733124-15476/_task_tmp.-ext-10000/_tmp.000004_0 to viewfs://<cluster_name>/user/<user_name>/.hive-staging_hive_2019-10-15_12-59-32_916_2461818728035733124-15476/_tmp.-ext-10000/000004_0
	at org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator.closeOp(AbstractFileMergeOperator.java:254)
	... 20 more

3、原因

初始化AbstractFileMergeOperator时，仅第一次更新taskId。

AbstractFileMergeOperator.java

private void updatePaths(Path tp, Path ttp) {
  if (taskId == null) {
    taskId = Utilities.getTaskId(jc);
  }

 这导致输出文件名的上述冲突。

4、解决方案

     set tez.am.container.reuse.enabled = false