HDFS FileSystem Shell
All FS shell commands take path URIs as arguments. The URI format is scheme://authority/path. For HDFS the scheme is hdfs, and for the Local FS the scheme is file. The scheme and authority are optional.
appendToFile
hdfs dfs -appendToFile <localsrc> ... <dst>
hdfs dfs -appendToFile localfile hdfs://nn.example.com/hadoop/hadoopfile
Returns 0 on success and 1 on error.
cat:Copies source paths to stdout.
hdfs dfs -cat URI [URI ...]
hdfs dfs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hdfs dfs -cat file:///file3 /user/hadoop/file4
Returns 0 on success and -1 on error.
chgrp:Change group association of files,改变文件所属的组,改变文件组权 change group
hdfs dfs -chgrp [-R] GROUP URI [URI ...]
The -R option will make the change recursively through the directory structure.
chmod:Change the permissions of files--change mode
hdfs dfs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]
chown:Change the owner of files
hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]
copyFromLocal:Similar to put command, except that the source is restricted to a local file reference.
hdfs dfs -copyFromLocal <localsrc> URI
The -f option will overwrite the destination if it already exists.
copyToLocal:Similar to get command, except that the destination is restricted to a local file reference.
hdfs dfs -copyToLocal [-ignorecrc] [-crc] URI <localdst>
count:Count the number of directories, files and bytes under the paths that match the specified file pattern
The output columns with -count are: DIR_COUNT, FILE_COUNT, CONTENT_SIZE FILE_NAME
The output columns with -count -q are: QUOTA, REMAINING_QUATA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAMEquota:配额;定额;限额
The -h option shows sizes in human readable format. h-humanity
hdfs dfs -count [-q] [-h] <paths>
hdfs dfs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hdfs dfs -count -q hdfs://nn1.example.com/file1
hdfs dfs -count -q -h hdfs://nn1.example.com/file1
Returns 0 on success and -1 on error.
cp:Copy files from source to destination. This command allows multiple sources as well in which case the destination must be a directory
hdfs dfs -cp [-f] [-p | -p[topax]] URI [URI ...] <dest>topax:timestamps, ownership, permission, ACL, XAttr ACL容许机舱负荷(Allowance Cabin Load)
hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2
hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
du:Displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.
hdfs dfs -du [-s] [-h] URI [URI ...]
The -s option will result in an aggregate summary of file lengths being displayed, rather than the individual files.
The -h option will format file sizes in a "human-readable" fashion (e.g 64.0m instead of 67108864)
dus:This command is deprecated. Instead use hdfs dfs -du -s.
expunge:Empty the Trash(垃圾).
get:Copy files to the local file system. Files that fail the CRC check may be copied with the -ignorecrc option. Files and CRCs may be copied using the -crc option.crc循环冗余码校验
hdfs dfs -get [-ignorecrc] [-crc] <src> <localdst>
hdfs dfs -get /user/hadoop/file localfile
hdfs dfs -get hdfs://nn.example.com/user/hadoop/file localfile
getfacl:Displays the Access Control Lists (ACLs:访问控制列表) of files and directories. If a directory has a default ACL, then getfacl also displays the default ACL.
hdfs dfs -getfacl [-R] <path>
hdfs dfs -getfacl /file
hdfs dfs -getfacl -R /dir
getfattr:Displays the extended attribute names and values (if any) for a file or directory
hdfs dfs -getfattr [-R] -n name | -d [-e en] <path>
hdfs dfs -getfattr -d /file
hdfs dfs -getfattr -R -n user.myAttr /dir
getmerge:Takes a source directory and a destination file as input and concatenates files in src into the destination local file.
hdfs dfs -getmerge <src> <localdst> [addnl]
ls:
hdfs dfs -ls [-R] <args>
hdfs dfs -ls /user/hadoop/file1
lsr:This command is deprecated. Instead use hdfs dfs -ls -R
mkdir:Takes path uri's as argument and creates directories.
moveFromLocal:Similar to put command, except that the source localsrc is deleted after it's copied.
hdfs dfs -moveFromLocal <localsrc> <dst>
moveToLocal:Displays a "Not implemented yet" message.
hdfs dfs -moveToLocal [-crc] <src> <dst>
mv:Moves files from source to destination.
hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2
put:Copy single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and writes to destination file system.
rm:Delete files specified as args.
hdfs dfs -rm [-f] [-r|-R] [-skipTrash] URI [URI ...]
hdfs dfs -rm hdfs://nn.example.com/file /user/hadoop/emptydir
All FS shell commands take path URIs as arguments. The URI format is scheme://authority/path. For HDFS the scheme is hdfs, and for the Local FS the scheme is file. The scheme and authority are optional.
appendToFile
hdfs dfs -appendToFile <localsrc> ... <dst>
hdfs dfs -appendToFile localfile hdfs://nn.example.com/hadoop/hadoopfile
Returns 0 on success and 1 on error.
cat:Copies source paths to stdout.
hdfs dfs -cat URI [URI ...]
hdfs dfs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hdfs dfs -cat file:///file3 /user/hadoop/file4
Returns 0 on success and -1 on error.
chgrp:Change group association of files,改变文件所属的组,改变文件组权 change group
hdfs dfs -chgrp [-R] GROUP URI [URI ...]
The -R option will make the change recursively through the directory structure.
chmod:Change the permissions of files--change mode
hdfs dfs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]
chown:Change the owner of files
hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]
copyFromLocal:Similar to put command, except that the source is restricted to a local file reference.
hdfs dfs -copyFromLocal <localsrc> URI
The -f option will overwrite the destination if it already exists.
copyToLocal:Similar to get command, except that the destination is restricted to a local file reference.
hdfs dfs -copyToLocal [-ignorecrc] [-crc] URI <localdst>
count:Count the number of directories, files and bytes under the paths that match the specified file pattern
The output columns with -count are: DIR_COUNT, FILE_COUNT, CONTENT_SIZE FILE_NAME
The output columns with -count -q are: QUOTA, REMAINING_QUATA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAMEquota:配额;定额;限额
The -h option shows sizes in human readable format. h-humanity
hdfs dfs -count [-q] [-h] <paths>
hdfs dfs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hdfs dfs -count -q hdfs://nn1.example.com/file1
hdfs dfs -count -q -h hdfs://nn1.example.com/file1
Returns 0 on success and -1 on error.
cp:Copy files from source to destination. This command allows multiple sources as well in which case the destination must be a directory
hdfs dfs -cp [-f] [-p | -p[topax]] URI [URI ...] <dest>topax:timestamps, ownership, permission, ACL, XAttr ACL容许机舱负荷(Allowance Cabin Load)
hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2
hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir
du:Displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.
hdfs dfs -du [-s] [-h] URI [URI ...]
The -s option will result in an aggregate summary of file lengths being displayed, rather than the individual files.
The -h option will format file sizes in a "human-readable" fashion (e.g 64.0m instead of 67108864)
dus:This command is deprecated. Instead use hdfs dfs -du -s.
expunge:Empty the Trash(垃圾).
get:Copy files to the local file system. Files that fail the CRC check may be copied with the -ignorecrc option. Files and CRCs may be copied using the -crc option.crc循环冗余码校验
hdfs dfs -get [-ignorecrc] [-crc] <src> <localdst>
hdfs dfs -get /user/hadoop/file localfile
hdfs dfs -get hdfs://nn.example.com/user/hadoop/file localfile
getfacl:Displays the Access Control Lists (ACLs:访问控制列表) of files and directories. If a directory has a default ACL, then getfacl also displays the default ACL.
hdfs dfs -getfacl [-R] <path>
hdfs dfs -getfacl /file
hdfs dfs -getfacl -R /dir
getfattr:Displays the extended attribute names and values (if any) for a file or directory
hdfs dfs -getfattr [-R] -n name | -d [-e en] <path>
hdfs dfs -getfattr -d /file
hdfs dfs -getfattr -R -n user.myAttr /dir
getmerge:Takes a source directory and a destination file as input and concatenates files in src into the destination local file.
hdfs dfs -getmerge <src> <localdst> [addnl]
ls:
hdfs dfs -ls [-R] <args>
hdfs dfs -ls /user/hadoop/file1
lsr:This command is deprecated. Instead use hdfs dfs -ls -R
mkdir:Takes path uri's as argument and creates directories.
moveFromLocal:Similar to put command, except that the source localsrc is deleted after it's copied.
hdfs dfs -moveFromLocal <localsrc> <dst>
moveToLocal:Displays a "Not implemented yet" message.
hdfs dfs -moveToLocal [-crc] <src> <dst>
mv:Moves files from source to destination.
hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2
put:Copy single src, or multiple srcs from local file system to the destination file system. Also reads input from stdin and writes to destination file system.
rm:Delete files specified as args.
hdfs dfs -rm [-f] [-r|-R] [-skipTrash] URI [URI ...]
hdfs dfs -rm hdfs://nn.example.com/file /user/hadoop/emptydir