HDFS基本操作-CSDN博客

本文链接：https://blog.csdn.net/baolibin528/article/details/43854291

HDFS基本操作

1、查看HDFS 目录：

执行命令：

hadoop fs -ls hdfs://192.168.1.100:9000/

[hadoop@baolibin ~]$ hadoop fs -lshdfs://192.168.1.100:9000/
Warning: $HADOOP_HOME is deprecated.
 
Found 1 items
drwxr-xr-x  - hadoop supergroup          02015-02-15 21:05 /usr
[hadoop@baolibin ~]$

2、对HDFS目录递归查看：

执行命令：

hadoop fs -lsr hdfs://192.168.1.100:9000/

 [hadoop@baolibin~]$ hadoop fs -lsr hdfs://192.168.1.100:9000/
Warning: $HADOOP_HOME is deprecated.
 
drwxr-xr-x  - hadoop supergroup          02015-02-15 21:05 /usr
drwxr-xr-x  - hadoop supergroup          02015-02-15 21:05 /usr/hadoop
drwxr-xr-x  - hadoop supergroup          02015-02-15 21:05 /usr/hadoop/tmp
drwxr-xr-x  - hadoop supergroup          02015-02-15 21:24 /usr/hadoop/tmp/mapred
drwx------  - hadoop supergroup          02015-02-15 21:24 /usr/hadoop/tmp/mapred/system
-rw-------  1 hadoop supergroup          42015-02-15 21:24 /usr/hadoop/tmp/mapred/system/jobtracker.info
[hadoop@baolibin ~]$

3、创建目录：

执行命令：

hadoop fs -mkdir input

[hadoop@baolibin ~]$ hadoop fs -mkdir input
Warning: $HADOOP_HOME is deprecated.
 
[hadoop@baolibin ~]$

[hadoop@baolibin ~]$ hadoop fs -ls
Warning: $HADOOP_HOME is deprecated.
 
Found 1 items
drwxr-xr-x  - hadoop supergroup          02015-02-15 22:09 /user/hadoop/input
[hadoop@baolibin ~]$

4、从Linux上传文件到HDFS：

执行命令：

hadoop fs -put /home/hadoop/baozi.txt /user/hadoop/input/

[hadoop@baolibin ~]$ hadoop fs -put/home/hadoop/baozi.txt /user/hadoop/input/
Warning: $HADOOP_HOME is deprecated.
 
[hadoop@baolibin ~]$

查看input 目录：

[hadoop@baolibin ~]$ hadoop fs -ls input
Warning: $HADOOP_HOME is deprecated.
 
Found 1 items
-rw-r--r--  1 hadoop supergroup         462015-02-15 22:13 /user/hadoop/input/baozi.txt
[hadoop@baolibin ~]$

5、从HDFS下载文件到Linux：

执行命令：

hadoop fs -get /user/hadoop/input/baozi.txt /home/hadoop/xiaobaozi

[hadoop@baolibin ~]$ hadoop fs -get/user/hadoop/input/baozi.txt /home/hadoop/xiaobaozi
Warning: $HADOOP_HOME is deprecated.
 
[hadoop@baolibin ~]$

查看：

[hadoop@baolibin ~]$ ll/home/hadoop/xiaobaozi
总用量 4
-rw-rw-r--. 1 hadoop hadoop 46 2月 15 22:17 baozi.txt
[hadoop@baolibin ~]$

6、查看文件内容：

执行命令：

hadoop fs -text input/baozi.txt

[hadoop@baolibin ~]$ hadoop fs -textinput/baozi.txt
Warning: $HADOOP_HOME is deprecated.
 
hadoop hello java baozi hbase hive hadoopjava
[hadoop@baolibin ~]$

7、删除文件：

执行命令：

hadoop fs -rm input/baozi.txt

[hadoop@baolibin ~]$ hadoop fs -rminput/baozi.txt
Warning: $HADOOP_HOME is deprecated.
 
Deletedhdfs://192.168.1.100:9000/user/hadoop/input/baozi.txt
[hadoop@baolibin ~]$

查看是否删除：

[hadoop@baolibin ~]$ hadoop fs -ls input/
Warning: $HADOOP_HOME is deprecated.
 
[hadoop@baolibin ~]$

8、递归删除文件：

执行命令：

hadoop fs -rmr input

[hadoop@baolibin ~]$ hadoop fs -rmr input
Warning: $HADOOP_HOME is deprecated.
 
Deletedhdfs://192.168.1.100:9000/user/hadoop/input
[hadoop@baolibin ~]$

查看是否删除：

[hadoop@baolibin ~]$ hadoop fs -ls
Warning: $HADOOP_HOME is deprecated.
 
[hadoop@baolibin ~]$

9、查看所有命令：

执行命令：

hadoop fs

[hadoop@baolibin ~]$ hadoop fs
Warning: $HADOOP_HOME is deprecated.
 
Usage: java FsShell
          [-ls <path>]
          [-lsr <path>]
          [-du <path>]
          [-dus <path>]
          [-count[-q] <path>]
          [-mv <src> <dst>]
          [-cp <src> <dst>]
          [-rm [-skipTrash] <path>]
          [-rmr [-skipTrash] <path>]
          [-expunge]
           [-put <localsrc> ... <dst>]
          [-copyFromLocal <localsrc> ... <dst>]
          [-moveFromLocal <localsrc> ... <dst>]
          [-get [-ignoreCrc] [-crc] <src> <localdst>]
          [-getmerge <src> <localdst> [addnl]]
          [-cat <src>]
          [-text <src>]
          [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
          [-moveToLocal [-crc] <src> <localdst>]
          [-mkdir <path>]
          [-setrep [-R] [-w] <rep> <path/file>]
          [-touchz <path>]
          [-test -[ezd] <path>]
          [-stat [format] <path>]
          [-tail [-f] <file>]
          [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
          [-chown [-R] [OWNER][:[GROUP]] PATH...]
          [-chgrp [-R] GROUP PATH...]
          [-help [cmd]]
 
Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|jobtracker:port>    specify a job tracker
-files <comma separated list offiles>    specify comma separatedfiles to be copied to the map reduce cluster
-libjars <comma separated list ofjars>    specify comma separated jarfiles to include in the classpath.
-archives <comma separated list ofarchives>    specify comma separatedarchives to be unarchived on the compute machines.
 
The general command line syntax is
bin/hadoop command [genericOptions][commandOptions]
 
[hadoop@baolibin ~]$

10、查看某个命令用法：

执行命令：

hadoop fs -help ls

[hadoop@baolibin ~]$ hadoop fs -help ls
Warning: $HADOOP_HOME is deprecated.
 
-ls <path>:     List the contents that match the specifiedfile pattern. If
                path is not specified, thecontents of /user/<currentUser>
                will be listed. Directoryentries are of the form
                        dirName (full path)<dir>
                and file entries are of theform
                        fileName(full path)<r n> size
                where n is the number ofreplicas specified for the file
                and size is the size of thefile, in bytes.
 
[hadoop@baolibin ~]$

11、查看HDFS 基本统计信息：

执行命令：

/usr/hadoop/bin/hadoopdfsadmin -report

[hadoop@baolibin current]$/usr/hadoop/bin/hadoop dfsadmin -report
Warning: $HADOOP_HOME is deprecated.
 
Configured Capacity: 7431069696 (6.92 GB)
Present Capacity: 2737635328 (2.55 GB)
DFS Remaining: 2737594368 (2.55 GB)
DFS Used: 40960 (40 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
 
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
 
Name: 192.168.1.100:50010
Decommission Status : Normal
Configured Capacity: 7431069696 (6.92 GB)
DFS Used: 40960 (40 KB)
Non DFS Used: 4693434368 (4.37 GB)
DFS Remaining: 2737594368(2.55 GB)
DFS Used%: 0%
DFS Remaining%: 36.84%
Last contact: Mon Feb 16 00:19:31 CST 2015
 
[hadoop@baolibin current]$

12、命令清单：

hadoop fs

      -help [cmd]      //显示命令的帮助信息
      -ls(r) <path>     //显示当前目录下所有文件
      -du(s) <path>    //显示目录中所有文件大小
      -count[-q] <path>     //显示目录中文件数量
      -mv <src> <dst>      //移动多个文件到目标目录
      -cp <src> <dst> //复制多个文件到目标目录
      -rm(r)              //删除文件(夹)
      -put <localsrc><dst>       //本地文件复制到hdfs
      -copyFromLocal       //同put
      -moveFromLocal      //从本地文件移动到hdfs
      -get [-ignoreCrc] <src><localdst>  //复制文件到本地，可以忽略crc校验
      -getmerge <src><localdst>            //将源目录中的所有文件排序合并到一个文件中
      -cat <src>  //在终端显示文件内容
      -text <src> //在终端显示文件内容
      -copyToLocal [-ignoreCrc]<src> <localdst>  //复制到本地
      -moveToLocal <src><localdst>
      -mkdir <path>  //创建文件夹
      -touchz <path>  //创建一个空文件

希望对初学者有所帮助。