[问题记录]hdfs删除文件提示rmr: Failed to move to trash

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/yeweiouyang/article/details/53518075

问题/现象

使用hadoop fs -rmr xxx删除hdfs上的文件失败

$ hadoop fs -rmr /app/lbs/nuomi-da-stat/stat_platform_auto/1004/dt=20161118/000000_0

rmr: Failed to move to trash: /app/lbs/nuomi-da-stat/stat_platform_auto/1004/dt=20161118/000000_0

原因分析

原因一:集群回收站空间占满
原因二:当前ugi没有权限删除对应的hdfs文件,通过hadoop fs -ls命令可以发现,用户权限是nuomi-abc,用户组权限是nuomi-da-stat,当前运行的ugi是在hadoop-site.xml文件中配置的,如果当前运行的ugi与用户权限或用户组权限不符合,就会因为没有权限执行而报错

$ hadoop fs -ls /app/lbs/nuomi-da-stat/stat_platform_auto/1405/dt=20161111/000000_0
Found 1 items
-rw-r--r--   3 nuomi-abc nuomi-da-stat         85 2016-11-18 05:09 /app/lbs/nuomi-da-stat/stat_platform_auto/1405/dt=20161111/000000_0

解决方案

原因一:集群回收站空间占满

方法:清空集群回收站

$ hadoop fs -rmr /user/nuomi-da-stat/.Trash/*

原因二:当前ugi没有权限删除对应的hdfs文件

方法一:更改hdfs文件权限

-chown [-R] [OWNER][:[GROUP]] PATH...
        Changes owner and group of a file.
        This is similar to shell's chown with a few exceptions.

    -R  modifies the files recursively. This is the only option
        currently supported.

        If only owner or group is specified then only owner or
        group is modified.

        The owner and group names may only cosists of digits, alphabet,
        and any of '-_.@/' i.e. [-_.@/a-zA-Z0-9]. The names are case
        sensitive.

        WARNING: Avoid using '.' to separate user name and group though
        Linux allows it. If user names have dots in them and you are
        using local file system, you might see surprising results since
        shell command 'chown' is used for local files.

如下命令,作用是递归地将/app/lbs/nuomi-da-stat/stat_platform_auto/下的所有文件,用户权限和组权限都改成nuomi-da-stat,文件多的话执行会很慢

hadoop fs -chown -R nuomi-da-stat:nuomi-da-stat /app/lbs/nuomi-da-stat/stat_platform_auto/

方法二:使用有权限的ugi来操作

$ hadoop fs -D hadoop.job.ugi=nuomi-abc -rmr /app/lbs/nuomi-da-stat/stat_platform_auto/1405/dt=20161111/000000_0
Moved to trash: /app/lbs/nuomi-da-stat/stat_platform_auto/1405/dt=20161111/000000_0
阅读更多
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页