05 测试hadoop自带词频统计demo

在了解了Hadoop中的存储组件HDFS之后，我们再来看一下Hadoop中另一个重要组件的计算MapReduce。HDFS搞定海量的存储，MapReduce搞定海量的计算。hadoop如其他优秀的开源组件一样，也提供了丰富的demo，下面我们就来看一下如何使用mapreduce自带demo进行词频统计。

1 系统、环境和约束条件

• 在CentOS7中安装hadoop并启动，作者的hadoop安装目录在/root/hadoop-2.5.2
  https://www.jianshu.com/p/b7ae3b51e559

2 操作

• 1 使用putty登录centos
• 2 执行以下命令：

# 切换到家目录
cd 
# 进入hadoop的bin目录
cd hadoop-2.5.2/bin
#  vim word，在其中加入以下内容并保存退出，读者可以随意加入别的内容，这是我们待会要统计词频的文件
hello i am zhangli
hello i am xiaoli
hi i am ali
who are you
i am xiaoli
# 上传word文件
./hdfs dfs -put word /word
# 查看上传结果
./hdfs dfs -cat /word
# 开始统计，其中
# ./yarn是执行命令
# jar是表示执行的是jar包
# /root/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar 表示要执行的jar包
# wordcount  是要执行过程的名字
# /word 是我们上传的待分析的文件在HDFS中的路径
# /output 是我们分析之后结果的输出路径
./yarn jar /root/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /word /output
# 等待一阵子，会有以下输出
19/05/30 12:29:41 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.100.192:8032
19/05/30 12:29:46 INFO input.FileInputFormat: Total input paths to process : 1
19/05/30 12:29:47 INFO mapreduce.JobSubmitter: number of splits:1
19/05/30 12:29:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559056674360_0002
19/05/30 12:29:50 INFO impl.YarnClientImpl: Submitted application application_1559056674360_0002
19/05/30 12:29:51 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1559056674360_0002/
19/05/30 12:29:51 INFO mapreduce.Job: Running job: job_1559056674360_0002
19/05/30 12:30:19 INFO mapreduce.Job: Job job_1559056674360_0002 running in uber mode : false
19/05/30 12:30:19 INFO mapreduce.Job:  map 0% reduce 0%
19/05/30 12:30:36 INFO mapreduce.Job:  map 100% reduce 0%
19/05/30 12:30:46 INFO mapreduce.Job:  map 100% reduce 100%
19/05/30 12:30:49 INFO mapreduce.Job: Job job_1559056674360_0002 completed successfully
19/05/30 12:30:50 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=111
        FILE: Number of bytes written=194141
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=156
        HDFS: Number of bytes written=65
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=15451
        Total time spent by all reduces in occupied slots (ms)=7614
        Total time spent by all map tasks (ms)=15451
        Total time spent by all reduce tasks (ms)=7614
        Total vcore-seconds taken by all map tasks=15451
        Total vcore-seconds taken by all reduce tasks=7614
        Total megabyte-seconds taken by all map tasks=15821824
        Total megabyte-seconds taken by all reduce tasks=7796736
    Map-Reduce Framework
        Map input records=5
        Map output records=17
        Map output bytes=135
        Map output materialized bytes=111
        Input split bytes=89
        Combine input records=17
        Combine output records=10
        Reduce input groups=10
        Reduce shuffle bytes=111
        Reduce input records=10
        Reduce output records=10
        Spilled Records=20
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=697
        CPU time spent (ms)=8200
        Physical memory (bytes) snapshot=445980672
        Virtual memory (bytes) snapshot=4215586816
        Total committed heap usage (bytes)=322437120
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=67
    File Output Format Counters 
        Bytes Written=65
# 查看/output输出，在以下路径中会看到有两个文件，其中_SUCCESS代表成功，part-r-00000代表输出结果
./hdfs dfs -ls /output
以下为输出：
Found 2 items
-rw-r--r--   2 root supergroup          0 2019-05-30 12:30 /output/_SUCCESS
-rw-r--r--   2 root supergroup         65 2019-05-30 12:30 /output/part-r-00000
# 查看词频统计结果
./hdfs dfs -cat /output/part-r-00000
# 以下为输出
ali 1
am  4
are 1
hello   1
hi  1
i   4
who 1
xiaoli  2
you 1
zhangli 1

以上就是利用hadoop自带的词频统计demo进行统计并查看统计结果的过程。