Hadoop基础知识（二）

最新推荐文章于 2021-08-12 10:13:58 发布

飞鸿踏雪泥-

最新推荐文章于 2021-08-12 10:13:58 发布

阅读量376

点赞数

分类专栏： hadoop 文章标签： hadoop command shell

本文链接：https://blog.csdn.net/java19930623/article/details/49186237

版权

hadoop 专栏收录该内容

8 篇文章 0 订阅

订阅专栏

Hadoop Shell介绍

bin目录下的Hadoop脚本是最基础的集群管理脚本，用户可以通过该脚本完成各种功能，如HDFS文件管理、MapReduce作业管理等。该脚本的使用方法为：
hadoop[–config confdir]COMMAND
其中，–config用于设置Hadoop配置文件目录，默认目录为
${HADOOP_HOME}/conf
而COMMAND是具体的某个命令。常用的有HDFS管理命令fs、作业管理命令job和作业提交命令jar等。

测试环境

1）HDFS测试
对HDFS文件系统进行查看文件，对文件或者文件的基本操作。（通过命令行的方式交互）

[hadoop1@H01 bin]$ pwd
/home/hadoop1/Hadoop/modules/hadoop-1.2.1/bin

[hadoop1@H01 bin]$ ./hadoop
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  mradmin              run a Map-Reduce admin client
  fsck                 run a DFS filesystem checking utility
  fs                   run a generic filesystem user client
  balancer             run a cluster balancing utility
  oiv                  apply the offline fsimage viewer to an fsimage
  fetchdt              fetch a delegation token from the NameNode
  jobtracker           run the MapReduce job Tracker node
  pipes                run a Pipes job
  tasktracker          run a MapReduce task Tracker node
  historyserver        run job history servers as a standalone daemon
  job                  manipulate MapReduce jobs
  queue                get information regarding JobQueues
  version              print the version
  jar <jar>            run a jar file
  distcp <srcurl> <desturl> copy file or directories recursively
  distcp2 <srcurl> <desturl> DistCp version 2
  archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
  classpath            prints the class path needed to get the
                       Hadoop jar and the required libraries
  daemonlog            get/set the log level for each daemon
 or
  CLASSNAME            run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

[hadoop1@H01 conf]$ pwd
/home/hadoop1/Hadoop/modules/hadoop-1.2.1/conf
[hadoop1@H01 conf]$ hadoop --config /home/hadoop1/Hadoop/modules/hadoop-1.2.1/conf/ fs
Usage: java FsShell
           [-ls <path>]
           [-lsr <path>]
           [-du <path>]
           [-dus <path>]
           [-count[-q] <path>]
           [-mv <src> <dst>]
           [-cp <src> <dst>]
           [-rm [-skipTrash] <path>]
           [-rmr [-skipTrash] <path>]
           [-expunge]
           [-put <localsrc> ... <dst>]
           [-copyFromLocal <localsrc> ... <dst>]
           [-moveFromLocal <localsrc> ... <dst>]
           [-get [-ignoreCrc] [-crc] <src> <localdst>]
           [-getmerge <src> <localdst> [addnl]]
           [-cat <src>]
           [-text <src>]
           [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
           [-moveToLocal [-crc] <src> <localdst>]
           [-mkdir <path>]
           [-setrep [-R] [-w] <rep> <path/file>]
           [-touchz <path>]
           [-test -[ezd] <path>]
           [-stat [format] <path>]
           [-tail [-f] <file>]
           [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
           [-chown [-R] [OWNER][:[GROUP]] PATH...]
           [-chgrp [-R] GROUP PATH...]
           [-help [cmd]]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|jobtracker:port>    specify a job tracker
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]