hive drop parition error when move the data of the partition to another namespace

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/houzhizhen/article/details/80323541

The error log on hive metastore.


2018-05-15 10:28:56,503 ERROR server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Error occurred during processing of message.
java.lang.IllegalArgumentException: Wrong FS: hdfs://ns2:8020/pathxxx0, expected: hdfs://ns1/pathxxx1
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:658)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
        at org.apache.hadoop.hdfs.DistributedFileSystem$43.doCall(DistributedFileSystem.java:2030)
        at org.apache.hadoop.hdfs.DistributedFileSystem$43.doCall(DistributedFileSystem.java:2026)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getEZForPath(DistributedFileSystem.java:2026)
        at org.apache.hadoop.hdfs.client.HdfsAdmin.getEncryptionZoneForPath(HdfsAdmin.java:310)
        at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1219)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.checkTrashPurgeCombination(HiveMetaStore.java:1584)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_partitions_req(HiveMetaStore.java:2785)
        at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy5.drop_partitions_req(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_partitions_req.getResult(ThriftHiveMetastore.java:9852)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_partitions_req.getResult(ThriftHiveMetastore.java:9836)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

The key problem is when create HdfsEncryptionShim object, the filesystem of hdfsAdmin is read from hive configuration.

public HdfsEncryptionShim(URI uri, Configuration conf) throws IOException {
            DistributedFileSystem dfs = (DistributedFileSystem)FileSystem.get(uri, conf);
            this.conf = conf;
            this.keyProvider = dfs.getClient().getKeyProvider();
            this.hdfsAdmin = new HdfsAdmin(uri, conf);
        }

When the namespace of hdfsAdmin is different from namespace of full path, the next sentence will throw file system not match exception. hdfsAdmin.getEncryptionZoneForPath(fullPath)


public boolean isPathEncrypted(Path path) throws IOException {
    Path fullPath;
    if(path.isAbsolute()) {
        fullPath = path;
    } else {
        fullPath = path.getFileSystem(this.conf).makeQualified(path);
    }

    return !"hdfs".equalsIgnoreCase(path.toUri().getScheme())?false:this.hdfsAdmin.getEncryptionZoneForPath(fullPath) != null;
}

阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页