hadoop mapred(hive)执行目录文件权限问题

最新推荐文章于 2024-09-26 08:51:43 发布

azhao_dn

最新推荐文章于 2024-09-26 08:51:43 发布

阅读量2.6w

点赞数 1

分类专栏： hadoop hive 文章标签： hadoop permissions access 作业 exception user

本文链接：https://blog.csdn.net/azhao_dn/article/details/6921398

版权

在配置其他linux用户使用hive 的cli客户端时，发现该用户没有对/tmp目录的写权限，于是将/tmp目录下权限设置为777：bin /hadoop fs -chmod -R 777 /tmp，然而第二天来时发现hadoop的文件rush任务失败，报错的日志信息如下：Job Submission failed with exception 'java.io.IOException(The ownership/permissions on the staging directory /tmp/hadoop-hadoop-user1/mapred/staging/hadoop-user1/.staging is not as expected. It is owned by hadoop-user1 and permissions are rwxrwxrwx. The directory must be owned by the submitter hadoop-user1 or by hadoop-user1 and permissions must be rwx------)
   原来hadoop mapred作业的运行用户在/tmp目录下创建了自己的作业空间：/tmp/hadoop-rsync，且该作业空间的权限必须为700，所以赶紧更改该目录权限： bin/hadoop fs -chmod -R 700 /tmp/hadoop-hadoop-user1 ，权限更改之后mapred作业运行正常。
   在修改之前，我将/tmp的目录修改为644，结果其他用户在使用hive是报错：
FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="tmp":hadoop-user1:supergroup:rw-rw-rw-)
java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="tmp":hadoop-user1:supergroup:rw-rw-rw-
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
        at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
        at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer