hadoop mapred(hive)执行目录 文件权限问题

hadoop 同时被 2 个专栏收录
64 篇文章 0 订阅
21 篇文章 0 订阅
在配置其他linux用户使用hive 的cli客户端时,发现该用户没有对/tmp目录的写权限,于是将/tmp目录下权限设置为777:bin  /hadoop fs -chmod -R 777 /tmp,然而第二天来时发现hadoop的文件rush任务失败,报错的日志信息如下:Job Submission failed with exception 'java.io.IOException(The ownership/permissions on the staging directory /tmp/hadoop-hadoop-user1/mapred/staging/hadoop-user1/.staging is not as expected. It is owned by hadoop-user1 and permissions are rwxrwxrwx. The directory must be owned by the submitter hadoop-user1 or by hadoop-user1 and permissions must be rwx------)
   原来hadoop mapred作业的运行用户在/tmp目录下创建了自己的作业空间:/tmp/hadoop-rsync,且该作业空间的权限必须为700,所以赶紧更改该目录权限: bin/hadoop fs -chmod -R 700 /tmp/hadoop-hadoop-user1 ,权限更改之后mapred作业运行正常。
   在修改之前,我将/tmp的目录修改为644,结果其他用户在使用hive是报错:
 FAILED: Hive Internal Error: java.lang.RuntimeException(org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="tmp":hadoop-user1:supergroup:rw-rw-rw-)
java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="tmp":hadoop-user1:supergroup:rw-rw-rw-
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:151)
        at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
        at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="tmp":hadoop-user1:supergroup:rw-rw-rw-
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
        at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:994)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:329)
        at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1050)
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:147)
        ... 15 more
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="tmp":hadoop-user1:supergroup:rw-rw-rw-
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:199)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:155)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:125)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4811)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:4790)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:1904)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:1886)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:716)
        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1383)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1379)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1377)

        at org.apache.hadoop.ipc.Client.call(Client.java:1030)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
        at $Proxy5.mkdirs(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at $Proxy5.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:992)
        ... 18 more
   在将/tmp 和/tmp/hive-wangqiang的全新修改为777后:
                          bin/hadoop fs -chmod 777 /tmp/hive-hadoop-user2
                          bin/hadoop fs -chmod 777 /tmp/
   错误信息为:Failed with exception java.io.IOException:org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="dw":hadoop-user1:supergroup:rw-r--r--
    最后将hive external表数据目录权限改为777后,查询才恢复正常。
    但是该用户在执行需要mapred的hql时,报错信息如下:
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop-user2, access=EXECUTE, inode="hadoop-hadoop-user1":hadoop-user1:supergroup:rwx------
   上述日志信息的意思是mapred作业在/tmp/目录下的文件夹hadoop-rsync权限拒绝用户wangqiang的执行访问。但是将该文件下面的权限更改到其他权限组合是,以rsync用户提交的mapred作业报错:permissions must be rwx------
    在仔细查看了/tmp/hadoop-rsync文件夹下面的目录结构后,发现了问题的关键所在:
[hadoop-user1@oser-624 hadoop-0.20.203.0]$ bin/hadoop fs -ls /tmp/hadoop-hadoop-user1/mapred/staging
Found 2 items
drwx------   - hadoop-user1     supergroup          0 2011-10-19 18:18 /tmp/hadoop-hadoop-user1/mapred/staging/hadoop-user1
drwx------   - hadoop-user2 supergroup          0 2011-10-27 18:38 /tmp/hadoop-hadoop-user1/mapred/staging/hadoop-user2
   原来不同用户提交的作业是在 /tmp/hadoop-hadoop-user1/mapred/staging/目录下以用户名区分,而之前的修改是直接使用-R选项直接修改/tmp/hadoop-rsync目录下的所有权限导致的错误,执行以下权限修改命令:
[hadoop-user1@oser-624 hadoop-0.20.203.0]$ bin/hadoop fs -chmod 777  /tmp/hadoop-hadoop-user1/mapred/
[hadoop-user1@oser-624 hadoop-0.20.203.0]$ bin/hadoop fs -chmod 777  /tmp/hadoop-hadoop-user1/mapred/staging
[hadoop-user1@oser-624 hadoop-0.20.203.0]$ bin/hadoop fs -chmod 777  /tmp/hadoop-hadoop-user1/
[hadoop-user1@oser-624 hadoop-0.20.203.0]$ bin/hadoop fs -chmod 777  /tmp
  hive查询正常。

    最近又遇上了这个问题,但是按照上面的方法修改后没能解决,于是查看namenode的日志:

2011-11-29 15:57:09,921 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9000, call mkdirs(/opt/data/hive-zhaoxiuxiang/hive_2011-11-29_15-57-08_094_4199830510252920639, rwxr-xr-x) from 192.168.1.187:18457: error: org.apache.hadoop.security.AccessControlException: Permission denied: user=zhaoxiuxiang, access=WRITE, inode="data":rsync:supergroup:rwxr-xr-x
    发现原来出现权限错误的是目录:/opt/data ,将该目录权限修改为777后错误解决。


附注:不同用户的mapred执行目录必须是700(rwx------)权限
  • 1
    点赞
  • 0
    评论
  • 0
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

相关推荐
©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值