Hadoop 1.x 伪分布式环境测试

       当Hadoop 1.x伪分布安装部署成功后，需要进行测试，包括HDFS的测试和MapReduce的wordcount，步骤如下：

1、创建HDFS文件目录

hadoop fs -mkdir /wc/
hadoop fs -mkdir /wc/input/

查看是否创建成功：

hadoop fs -lsr /

2、上传文件

     上传测试文件到上述input文件目录中。

hadoop fs -put /opt/modules/hadoop-1.2.1/conf/*.xml /wc/input/

查看是否上传成功：

hadoop fs -lsr /wc/input/

3、运行wordcount

hadoop jar hadoop-examples-1.2.1.jar wordcount /wc/input/ /wc/output/

中/wc/input为文件系统的输入目录；/wc/output/为输出目录，不能存在，否则报错。

运行结果如下：

[root@hadoop-master hadoop-1.2.1]# hadoop jar hadoop-examples-1.2.1.jar wordcount /wc/input/ /wc/output/
16/02/28 17:44:17 INFO input.FileInputFormat: Total input paths to process : 7
16/02/28 17:44:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library
16/02/28 17:44:17 WARN snappy.LoadSnappy: Snappy native library not loaded
16/02/28 17:44:18 INFO mapred.JobClient: Running job: job_201602281721_0001
16/02/28 17:44:20 INFO mapred.JobClient:  map 0% reduce 0%
16/02/28 17:44:45 INFO mapred.JobClient:  map 14% reduce 0%
16/02/28 17:44:47 INFO mapred.JobClient:  map 28% reduce 0%
16/02/28 17:44:54 INFO mapred.JobClient:  map 42% reduce 0%
16/02/28 17:44:56 INFO mapred.JobClient:  map 57% reduce 0%
16/02/28 17:45:02 INFO mapred.JobClient:  map 71% reduce 0%
16/02/28 17:45:03 INFO mapred.JobClient:  map 85% reduce 0%
16/02/28 17:45:04 INFO mapred.JobClient:  map 85% reduce 19%
16/02/28 17:45:06 INFO mapred.JobClient:  map 100% reduce 19%
16/02/28 17:45:11 INFO mapred.JobClient:  map 100% reduce 100%
16/02/28 17:45:12 INFO mapred.JobClient: Job complete: job_201602281721_0001
16/02/28 17:45:12 INFO mapred.JobClient: Counters: 29
16/02/28 17:45:12 INFO mapred.JobClient:   Job Counters 
16/02/28 17:45:12 INFO mapred.JobClient:     Launched reduce tasks=1
16/02/28 17:45:12 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=81748
16/02/28 17:45:12 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
16/02/28 17:45:12 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
16/02/28 17:45:12 INFO mapred.JobClient:     Launched map tasks=7
16/02/28 17:45:12 INFO mapred.JobClient:     Data-local map tasks=7
16/02/28 17:45:12 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=24763
16/02/28 17:45:12 INFO mapred.JobClient:   File Output Format Counters 
16/02/28 17:45:12 INFO mapred.JobClient:     Bytes Written=6578
16/02/28 17:45:12 INFO mapred.JobClient:   FileSystemCounters
16/02/28 17:45:12 INFO mapred.JobClient:     FILE_BYTES_READ=10629
16/02/28 17:45:12 INFO mapred.JobClient:     HDFS_BYTES_READ=16490
16/02/28 17:45:12 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=452276
16/02/28 17:45:12 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=6578
16/02/28 17:45:12 INFO mapred.JobClient:   File Input Format Counters 
16/02/28 17:45:12 INFO mapred.JobClient:     Bytes Read=15594
16/02/28 17:45:12 INFO mapred.JobClient:   Map-Reduce Framework
16/02/28 17:45:12 INFO mapred.JobClient:     Map output materialized bytes=10665
16/02/28 17:45:12 INFO mapred.JobClient:     Map input records=388
16/02/28 17:45:12 INFO mapred.JobClient:     Reduce shuffle bytes=10665
16/02/28 17:45:12 INFO mapred.JobClient:     Spilled Records=1196
16/02/28 17:45:12 INFO mapred.JobClient:     Map output bytes=21323
16/02/28 17:45:12 INFO mapred.JobClient:     CPU time spent (ms)=37490
16/02/28 17:45:12 INFO mapred.JobClient:     Total committed heap usage (bytes)=1003384832
16/02/28 17:45:12 INFO mapred.JobClient:     Combine input records=1761
16/02/28 17:45:12 INFO mapred.JobClient:     SPLIT_RAW_BYTES=896
16/02/28 17:45:12 INFO mapred.JobClient:     Reduce input records=598
16/02/28 17:45:12 INFO mapred.JobClient:     Reduce input groups=427
16/02/28 17:45:12 INFO mapred.JobClient:     Combine output records=598
16/02/28 17:45:12 INFO mapred.JobClient:     Physical memory (bytes) snapshot=1320984576
16/02/28 17:45:12 INFO mapred.JobClient:     Reduce output records=427
16/02/28 17:45:12 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=5717614592
16/02/28 17:45:12 INFO mapred.JobClient:     Map output records=1761

通过http://192.168.1.180:50030/jobtracker.jsp可以看到具体的job任务

通过 http://192.168.1.180:50070/dfshealth.jsp可以看到运行结果

当然，通过如下命令也可以查看运行结果：

hadoop fs -text  /wc/output/part-r-00000

或者通过如下命令查看：

hadoop fs -cat  /wc/output/part-r-00000

        通过上述步骤，可以验证安装是否成功。