Hadoop3.0.3集群自带WordCount程序测试MapRedece

Hadoop3.0.3自带的实例程序路径在
/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar中

在HDFS目录中创建input目录

[root@master]# hdfs dfs -mkdir /input
[root@master hadoop]# hdfs dfs -rm -r /input  #删除HDFS中的input目录

1.把测试文件上传到HDFS目录中

此处以/usr/local/hadoop/README.txt为例，把README.txt文件上传到HDFS的/input目录中

[root@master hadoop]# hdfs dfs -put README.txt /input

上传后可用如下命令看到。

[root@master hadoop]# hdfs dfs -ls /input
2018-11-26 20:05:11,067 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   3 root supergroup       1366 2018-11-26 20:02 /input/README.txt
[root@master hadoop]# 

也可在Web端看到。

2.使用自带wordcount程序测试MapRedece

[root@master hadoop]# hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount /input /output

执行过程如下：

2018-11-29 15:56:27,523 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-11-29 15:56:28,329 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.86.245:8050
2018-11-29 15:56:28,778 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1543478019442_0001
2018-11-29 15:56:29,894 INFO input.FileInputFormat: Total input files to process : 1
2018-11-29 15:56:30,081 INFO mapreduce.JobSubmitter: number of splits:1
2018-11-29 15:56:30,124 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-11-29 15:56:30,259 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1543478019442_0001
2018-11-29 15:56:30,261 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-11-29 15:56:30,458 INFO conf.Configuration: resource-types.xml not found
2018-11-29 15:56:30,458 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-11-29 15:56:30,686 INFO impl.YarnClientImpl: Submitted application application_1543478019442_0001
2018-11-29 15:56:30,720 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1543478019442_0001/
2018-11-29 15:56:30,720 INFO mapreduce.Job: Running job: job_1543478019442_0001
2018-11-29 15:56:38,883 INFO mapreduce.Job: Job job_1543478019442_0001 running in uber mode : false
2018-11-29 15:56:38,886 INFO mapreduce.Job:  map 0% reduce 0%
2018-11-29 15:56:44,985 INFO mapreduce.Job:  map 100% reduce 0%
2018-11-29 15:56:50,034 INFO mapreduce.Job:  map 100% reduce 100%
2018-11-29 15:56:51,060 INFO mapreduce.Job: Job job_1543478019442_0001 completed successfully
2018-11-29 15:56:51,177 INFO mapreduce.Job: Counters: 53
.......

3.列出输出目录

[root@master ~]# hadoop fs -ls /output
2018-11-27 14:44:13,654 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   3 root supergroup          0 2018-11-27 14:27 /output/_SUCCESS
-rw-r--r--   3 root supergroup       1306 2018-11-27 14:27 /output/part-r-00000
[root@master ~]# hdfs dfs -ls /output   #或此命令

4.列出输出结果

[root@master ~]# hdfs dfs -cat /output/part-r-00000
[root@master ~]# hadoop fs -cat /output/part-r-00000

5.Hadoop常用命令

hadoop fs -mkdir /tmp/input              在HDFS上新建文件夹
hadoop fs -put input1.txt /tmp/input  把本地文件input1.txt传到HDFS的/tmp/input目录下
hadoop fs -get  input1.txt /tmp/input/input1.txt  把HDFS文件拉到本地
hadoop fs -ls /tmp/output                  列出HDFS的某目录
hadoop fs -cat /tmp/ouput/output1.txt  查看HDFS上的文件
hadoop fs -rmr /home/less/hadoop/tmp/output  删除HDFS上的目录
hadoop dfsadmin -report 查看HDFS状态，比如有哪些datanode，每个datanode的情况
hadoop dfsadmin -safemode leave  离开安全模式
hadoop dfsadmin -safemode enter  进入安全模式