分区修复失败的问题 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

分区修复失败的问题 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

一、问题

在把CDH集群上的数据迁移到apache集群上时，遇到错误，记录一下。

1.脚本执行的时候报错

INFO  : Compiling command(queryId=hadoop_20211123141935_050bb456-601d-4f8d-bb3c-0b274461b4f8): msck repair table xxx_database.dwd_xxx_log
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hadoop_20211123141935_050bb456-601d-4f8d-bb3c-0b274461b4f8); Time taken: 0.043 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hadoop_20211123141935_050bb456-601d-4f8d-bb3c-0b274461b4f8): msck repair table xxx_database.dwd_xxx_log
INFO  : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
INFO  : Completed executing command(queryId=hadoop_20211123141935_050bb456-601d-4f8d-bb3c-0b274461b4f8); Time taken: 0.043 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)

通过查看我们得知，数据迁移完成后，修复表分区的时候，出现了错误。

2.分区修复时报错

打开hive客户端，验证。

hive (log_collection)> 
                     > msck repair table log_collection.dwd_webclick_log;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

二、解决

hive (log_collection)> set hive.msck.path.validation=ignore;
hive (log_collection)> msck repair table log_collection.dwd_webclick_log;
...

epair: Added partition to metastore dwd_webclick_log:dt=2021-05-02
Repair: Added partition to metastore dwd_webclick_log:dt=2021-03-28
Repair: Added partition to metastore dwd_webclick_log:dt=2021-07-24
Repair: Added partition to metastore dwd_webclick_log:dt=2021-09-09
Repair: Added partition to metastore dwd_webclick_log:dt=2021-04-26
Repair: Added partition to metastore dwd_webclick_log:dt=2021-10-15
Time taken: 1.617 seconds, Fetched: 265 row(s)

修改数据迁移脚本，在执行分区修复前，设置hive参数。

已经验证执行。