hadoop3.x——直接操作（shell）

相关知识

调用文件系统(FS)Shell命令应使用 hadoop fs <args>的形式。 所有的的FS shell命令使用URI路径作为参数。URI格式是scheme://authority/path。对HDFS文件系统，scheme是hdfs，对本地文件系统，scheme是file。其中scheme和authority参数都是可选的，如果未加指定，就会使用配置中指定的默认scheme。一个HDFS文件或目录比如/parent/child可以表示成hdfs://namenode:namenodeport/parent/child，或者更简单的/parent/child（假设你配置文件中的默认值是namenode:namenodeport）。大多数FS Shell命令的行为和对应的Unix Shell命令类似，出错信息会输出到stderr，其他信息输出到stdout。

常用的hdfs操作

1. -ls——显示目录信息

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -ls /
   Found 5 items
   drwxr-xr-x   - adming supergroup          0 2022-09-04 16:52 /sanguo
   drwxrwx---   - adming supergroup          0 2022-09-02 15:49 /tmp
   drwxr-xr-x   - adming supergroup          0 2022-08-21 17:17 /wcinput
   drwxr-xr-x   - adming supergroup          0 2022-09-02 15:50 /wcoutput1
   drwxr-xr-x   - adming supergroup          0 2022-08-21 17:47 /wcoutput2
   [adming@hadoop101 hadoop-3.1.3]$ 
   

2. -cat——显示文件内容

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -cat /sanguo/wuguo.txt
   2022-09-04 17:36:06,120 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
   wuguo
   

3. -chgrp、chmod、-chown：Linux文件系统中的用法一样，修改文件所属权限

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -chmod 666 /sanguo/wuguo.txt
   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -chown adming:adming /sanguo/wuguo.txt
   

   

4. -mkdir——创建路径

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -mkdir /jinguo
   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -ls /
   Found 6 items
   drwxr-xr-x   - adming supergroup          0 2022-09-04 17:38 /jinguo
   drwxr-xr-x   - adming supergroup          0 2022-09-04 16:52 /sanguo
   drwxrwx---   - adming supergroup          0 2022-09-02 15:49 /tmp
   drwxr-xr-x   - adming supergroup          0 2022-08-21 17:17 /wcinput
   drwxr-xr-x   - adming supergroup          0 2022-09-02 15:50 /wcoutput1
   drwxr-xr-x   - adming supergroup          0 2022-08-21 17:47 /wcoutput2
   

5. -cp——从HDFS的一个路径拷贝到另一个HDFS路径

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -cp /sanguo/shuguo.txt /jinguo
   2022-09-04 17:42:04,294 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
   2022-09-04 17:42:06,179 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -ls /jinguo
   Found 1 items
   -rw-r--r--   3 adming supergroup         14 2022-09-04 17:42 /jinguo/shuguo.txt
   

6. -mv——在HDFS目录中移动文件

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -ls /jinguo
   Found 1 items
   -rw-r--r--   3 adming supergroup         14 2022-09-04 17:42 /jinguo/shuguo.txt
   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -mv /sanguo/weiguo.txt /jinguo
   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -ls /jinguo
   Found 2 items
   -rw-r--r--   3 adming supergroup         14 2022-09-04 17:42 /jinguo/shuguo.txt
   -rw-r--r--   3 adming supergroup          7 2022-09-04 16:48 /jinguo/weiguo.txt
   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -ls /sanguo
   Found 2 items
   -rw-r--r--   3 adming supergroup         14 2022-09-04 17:08 /sanguo/shuguo.txt
   -rw-rw-rw-   3 adming adming              6 2022-09-04 16:52 /sanguo/wuguo.txt
   

7. -tail——显示文件末尾1kb的数据

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -tail /sanguo/shuguo.txt
   2022-09-04 17:46:47,984 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
   liubei
   liubei

8. -rm——删除文件或文件夹

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -rm /sanguo/shuguo.txt
   Deleted /sanguo/shuguo.txt
   

9. -rm -r：递归删除文件夹中的所有内容

   [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -rm -r /sanguo
   Deleted /sanguo
   

   
10. -du——统计文件夹的大小信息

    [adming@hadoop101 hadoop-3.1.3]$ hadoop fs -du /jinguo
    14  42  /jinguo/shuguo.txt
    7   21  /jinguo/weiguo.txt