Hadoop3.0.3自带的实例程序路径在
/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar
中
在HDFS目录中创建input目录
[root@master]# hdfs dfs -mkdir /input
[root@master hadoop]# hdfs dfs -rm -r /input #删除HDFS中的input目录
1.把测试文件上传到HDFS目录中
此处以/usr/local/hadoop/README.txt
为例,把README.txt
文件上传到HDFS的/input
目录中
[root@master hadoop]# hdfs dfs -put README.txt /input
上传后可用如下命令看到。
[root@master hadoop]# hdfs dfs -ls /input
2018-11-26 20:05:11,067 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r-- 3 root supergroup 1366 2018-11-26 20:02 /input/README.txt
[root@master hadoop]#
也可在Web端看到。
2.使用自带wordcount程序测试MapRedece
[root@master hadoop]# hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount /input /output
执行过程如下:
2018-11-29 15:56:27,523 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-11-29 15:56:28,329 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.86.245:8050
2018-11-29 15:56:28,778 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1543478019442_0001
2018-11-29 15:56:29,894 INFO input.FileInputFormat: Total input files to process : 1
2018-11-29 15:56:30,081 INFO mapreduce.JobSubmitter: number of splits:1
2018-11-29 15:56:30,124 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-11-29 15:56:30,259 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1543478019442_0001
2018-11-29 15:56:30,261 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-11-29 15:56:30,458 INFO conf.Configuration: resource-types.xml not found
2018-11-29 15:56:30,458 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-11-29 15:56:30,686 INFO impl.YarnClientImpl: Submitted application application_1543478019442_0001
2018-11-29 15:56:30,720 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1543478019442_0001/
2018-11-29 15:56:30,720 INFO mapreduce.Job: Running job: job_1543478019442_0001
2018-11-29 15:56:38,883 INFO mapreduce.Job: Job job_1543478019442_0001 running in uber mode : false
2018-11-29 15:56:38,886 INFO mapreduce.Job: map 0% reduce 0%
2018-11-29 15:56:44,985 INFO mapreduce.Job: map 100% reduce 0%
2018-11-29 15:56:50,034 INFO mapreduce.Job: map 100% reduce 100%
2018-11-29 15:56:51,060 INFO mapreduce.Job: Job job_1543478019442_0001 completed successfully
2018-11-29 15:56:51,177 INFO mapreduce.Job: Counters: 53
.......
3.列出输出目录
[root@master ~]# hadoop fs -ls /output
2018-11-27 14:44:13,654 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 3 root supergroup 0 2018-11-27 14:27 /output/_SUCCESS
-rw-r--r-- 3 root supergroup 1306 2018-11-27 14:27 /output/part-r-00000
[root@master ~]# hdfs dfs -ls /output #或此命令
4.列出输出结果
[root@master ~]# hdfs dfs -cat /output/part-r-00000
[root@master ~]# hadoop fs -cat /output/part-r-00000
5.Hadoop常用命令
hadoop fs -mkdir /tmp/input 在HDFS上新建文件夹
hadoop fs -put input1.txt /tmp/input 把本地文件input1.txt传到HDFS的/tmp/input目录下
hadoop fs -get input1.txt /tmp/input/input1.txt 把HDFS文件拉到本地
hadoop fs -ls /tmp/output 列出HDFS的某目录
hadoop fs -cat /tmp/ouput/output1.txt 查看HDFS上的文件
hadoop fs -rmr /home/less/hadoop/tmp/output 删除HDFS上的目录
hadoop dfsadmin -report 查看HDFS状态,比如有哪些datanode,每个datanode的情况
hadoop dfsadmin -safemode leave 离开安全模式
hadoop dfsadmin -safemode enter 进入安全模式