HDFS常用命令行操作

最新推荐文章于 2023-11-06 21:00:43 发布

时光追溯者

最新推荐文章于 2023-11-06 21:00:43 发布

阅读量233

点赞数 1

分类专栏：我的Hadoop学习之旅文章标签： hdfs

本文链接：https://blog.csdn.net/weixin_43540084/article/details/117826825

版权

我的Hadoop学习之旅专栏收录该内容

8 篇文章 0 订阅

订阅专栏

本文详细介绍了HDFS的命令行操作，包括查看文件和目录的`ls`命令，上传本地文件到HDFS的`put`命令，以及移动文件的`moveFromLocal`和下载文件的`get`命令。此外，还提到了`appendToFile`用于追加文件内容到HDFS文件的功能。请注意，使用这些命令时需要确保用户具有相应的权限。

摘要由CSDN通过智能技术生成

HDFS命令行操作

查看HDFS所有支持命令列表

$ hdfs dfs
Usage: hadoop fs [generic options]
	[-appendToFile <localsrc> ... <dst>]
	[-cat [-ignoreCrc] <src> ...]
	[-checksum <src> ...]
	[-chgrp [-R] GROUP PATH...]
	[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
	[-chown [-R] [OWNER][:[GROUP]] PATH...]
	[-copyFromLocal [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
	[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
	[-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] <path> ...]
	[-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]
	[-createSnapshot <snapshotDir> [<snapshotName>]]
	[-deleteSnapshot <snapshotDir> <snapshotName>]
	[-df [-h] [<path> ...]]
	[-du [-s] [-h] [-x] <path> ...]
	[-expunge]
	[-find <path> ... <expression> ...]
	[-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
	[-getfacl [-R] <path>]
	[-getfattr [-R] {-n name | -d} [-e en] <path>]
	[-getmerge [-nl] [-skip-empty-file] <src> <localdst>]
	[-help [cmd ...]]
	[-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
	[-mkdir [-p] <path> ...]
	[-moveFromLocal <localsrc> ... <dst>]
	[-moveToLocal <src> <localdst>]
	[-mv <src> ... <dst>]
	[-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>]
	[-renameSnapshot <snapshotDir> <oldName> <newName>]
	[-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
	[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
	[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
	[-setfattr {-n name [-v value] | -x name} <path>]
	[-setrep [-R] [-w] <rep> <path> ...]
	[-stat [format] <path> ...]
	[-tail [-f] <file>]
	[-test -[defsz] <path>]
	[-text [-ignoreCrc] <src> ...]
	[-touchz <path> ...]
	[-truncate [-w] <length> <path> ...]
	[-usage [cmd ...]]

Generic options supported are:
-conf <configuration file>        specify an application configuration file
-D <property=value>               define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port>  specify a ResourceManager
-files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machines

The general command line syntax is:
command [genericOptions] [commandOptions]

ls命令：查看HDFS系统中的文件和目录。

$ hdfs fs -ls /
Found 4 items
drwxr-xr-x   - hadoop supergroup          0 2021-06-08 16:13 /input
drwxr-xr-x   - hadoop supergroup          0 2021-06-08 16:19 /output
drwx------   - hadoop supergroup          0 2021-06-08 16:19 /tmp
drwxr-xr-x   - hadoop supergroup          0 2021-03-03 10:35 /user

$ hdfs fs -ls -R /  #递归列出HDFS文件系统根目录下的所有目录和文件
drwxr-xr-x   - hadoop supergroup          0 2021-06-08 16:13 /input
-rw-r--r--   1 hadoop supergroup       1366 2021-06-08 16:13 /input/README.txt
drwxr-xr-x   - hadoop supergroup          0 2021-06-08 16:19 /output
-rw-r--r--   1 hadoop supergroup          0 2021-06-08 16:19 /output/_SUCCESS
-rw-r--r--   1 hadoop supergroup       9456 2021-06-08 16:19 /output/part-r-00000
drwx------   - hadoop supergroup          0 2021-06-08 16:19 /tmp
drwx------   - hadoop supergroup          0 2021-03-03 21:13 /tmp/hadoop-yarn
drwx------   - hadoop supergroup          0 2021-03-03 21:13 /tmp/hadoop-yarn/staging
drwx------   - hadoop supergroup          0 2021-03-03 21:13 /tmp/hadoop-yarn/staging/hadoop
drwx------   - hadoop supergroup          0 2021-06-08 16:19 /tmp/hadoop-yarn/staging/hadoop/.staging
drwxr-xr-x   - hadoop supergroup          0 2021-03-03 21:26 /tmp/hadoop-yarn/staging/history
drwxrwx---   - hadoop supergroup          0 2021-03-03 21:26 /tmp/hadoop-yarn/staging/history/done
drwxrwx---   - hadoop supergroup          0 2021-03-03 21:26 /tmp/hadoop-yarn/staging/history/done/2021
drwxrwx---   - hadoop supergroup          0 2021-03-03 21:26 /tmp/hadoop-yarn/staging/history/done/2021/03
drwxrwx---   - hadoop supergroup          0 2021-03-03 21:26 /tmp/hadoop-yarn/staging/history/done/2021/03/03
drwxrwx---   - hadoop supergroup          0 2021-03-03 21:26 /tmp/hadoop-yarn/staging/history/done/2021/03/03/000000
-rwxrwx---   1 hadoop supergroup      33826 2021-03-03 21:13 /tmp/hadoop-yarn/staging/history/done/2021/03/03/000000/job_1614776397653_0001-1614777200689-hadoop-word+count-1614777216046-1-1-SUCCEEDED-default-1614777206362.jhist
-rwxrwx---   1 hadoop supergroup     206104 2021-03-03 21:13 /tmp/hadoop-yarn/staging/history/done/2021/03/03/000000/job_1614776397653_0001_conf.xml
drwxrwxrwt   - hadoop supergroup          0 2021-03-03 21:13 /tmp/hadoop-yarn/staging/history/done_intermediate
drwxrwx---   - hadoop supergroup          0 2021-06-08 16:19 /tmp/hadoop-yarn/staging/history/done_intermediate/hadoop
-rwxrwx---   1 hadoop supergroup      33841 2021-06-08 16:19 /tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1623139763489_0001-1623140353754-hadoop-word+count-1623140370783-1-1-SUCCEEDED-default-1623140360282.jhist
-rwxrwx---   1 hadoop supergroup        349 2021-06-08 16:19 /tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1623139763489_0001.summary
-rwxrwx---   1 hadoop supergroup     206086 2021-06-08 16:19 /tmp/hadoop-yarn/staging/history/done_intermediate/hadoop/job_1623139763489_0001_conf.xml
drwxrwxrwt   - hadoop hadoop              0 2021-06-08 16:19 /tmp/logs
drwxrwx---   - hadoop hadoop              0 2021-06-08 16:19 /tmp/logs/hadoop
drwxrwx---   - hadoop hadoop              0 2021-06-08 16:19 /tmp/logs/hadoop/logs
drwxrwx---   - hadoop hadoop              0 2021-06-08 16:19 /tmp/logs/hadoop/logs/application_1623139763489_0001
-rw-r-----   1 hadoop hadoop         114561 2021-06-08 16:19 /tmp/logs/hadoop/logs/application_1623139763489_0001/zbq-Lenovo-Legion-R7000-2020_34751
drwxr-xr-x   - hadoop supergroup          0 2021-03-03 10:35 /user
drwxr-xr-x   - hadoop supergroup          0 2021-03-03 21:13 /user/hadoop
drwxr-xr-x   - hadoop supergroup          0 2021-03-03 10:50 /user/hadoop/input
-rw-r--r--   1 hadoop supergroup      15841 2021-03-03 10:50 /user/hadoop/input/NOTICE.txt
drwxr-xr-x   - hadoop supergroup          0 2021-03-03 10:52 /user/hadoop/output
-rw-r--r--   1 hadoop supergroup          0 2021-03-03 10:52 /user/hadoop/output/_SUCCESS
-rw-r--r--   1 hadoop supergroup       9456 2021-03-03 10:52 /user/hadoop/output/part-r-00000
drwxr-xr-x   - hadoop supergroup          0 2021-03-03 21:13 /user/hadoop/output2
-rw-r--r--   1 hadoop supergroup          0 2021-03-03 21:13 /user/hadoop/output2/_SUCCESS
-rw-r--r--   1 hadoop supergroup       9456 2021-03-03 21:13 /user/hadoop/output2/part-r-00000

put命令：将本地文件上传到HDFS文件中

$ hdfs dfs -put 1.copy.txt /input

moveFromLocal: 将本地文件移动到HDFS系统中，可一次移动多个文件，并且在移动完毕后将源文件删除

$ $ hdfs dfs -moveFromLocal 2.txt /input/

get命令：将HDFS系统中的文件下载到本地，下载文件不能与本地文件名相同，下载多个文件时要先创建文件夹

$ hdfs dfs -get /input/README.txt  README.txt #单个文件
$ hdfs dfs -get /input/README.txt  /input/1.copy.txt /input/2.txt getFromHDFS #多个文件

需要注意的时用户要对当前目录有可写权限

drwxrwxr-x 2 hadoop hadoop 4096 Jun 11 22:33 getFromHDFS

rm、mkdir、cp、mv等命令与Linux shell命令大体类似，只是必须添加hdfs dfs前缀

appendToFile: 将单个或多个文件的内容从本地系统追加到HDFS系统的文件中。若要一次追加多个本地系统文件中的内容，需要用空格隔开

$ hdfs dfs -appendToFile 1.copy.txt  /input/2.txt

$ $ hdfs dfs -appendToFile 1.copy.txt README.txt  /input/2.txt

时光追溯者

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫

专栏目录