09 使用python完成词频统计

1 系统、软件以及前提约束

• CentOS-7 64
  为减少linux权限对初学者造成影响，所有命令均在linux的root权限下进行操作。
• 已安装hadoop-2.5.2 https://www.jianshu.com/p/5707c5ccd85b
• CentOS7当中已经默认安装python3.7.3

2 操作步骤

• 创建mapper.py文件

#!/usr/bin/python

import sys

# input comes from STDIN (standard input)
for line in sys.stdin:
    # remove leading and trailing whitespace
    line = line.strip()
    # split the line into words
    words = line.split()
    # increase counters
    for word in words:
        # write the results to STDOUT (standard output);
        # what we output here will be the input for the
        # Reduce step, i.e. the input for reducer.py
        #
        # tab-delimited; the trivial word count is 1
        print ('%s\t%s' % (word, 1))

验证，执行以下语句：

echo aa bb cc dd aa cc|python mapper.py

得到以下结果：

查看统计结果

• 创建reducer.py文件：

#!/usr/bin/python

from operator import itemgetter
import sys

current_word = None
current_count = 0
word = None

# input comes from STDIN
for line in sys.stdin:
    # remove leading and trailing whitespace
    line = line.strip()

    # parse the input we got from mapper.py
    word, count = line.split('\t', 1)

    # convert count (currently a string) to int
    try:
        count = int(count)
    except ValueError:
        # count was not a number, so silently
        # ignore/discard this line
        continue

    # this IF-switch only works because Hadoop sorts map output
    # by key (here: word) before it is passed to the reducer
    if current_word == word:
        current_count += count
    else:
        if current_word:
            # write result to STDOUT
            print ('%s\t%s' % (current_word, current_count))
        current_count = count
        current_word = word

# do not forget to output the last word if needed!
if current_word == word:
    print ('%s\t%s' % (current_word, current_count))

验证，执行以下语句：

echo aa bb cc dd aa cc|python mapper.py|sort|python reducer.py

得到以下结果：

查看统计结果

• 创建一个文件info.txt，内容如下：

aa bb cc dd aa cc
aa bb cc dd aa cc
aa bb cc dd aa cc
aa bb cc dd aa cc
aa bb cc dd aa cc cc dd

• 上传该文件到HDFS的/data的info文件中

hdfs dfs -mkdir /data
hdfs dfs -put info.txt /data/info

• 执行以下命令，确保hdfs下/out99不存在

$HADOOP_HOME/bin/hadoop jar 
 $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.5.2.jar 
 -input "/data/*" 
 -output "/out99" 
 -mapper "python mapper.py" 
 -reducer "python reducer.py" 
 -file "/root/mapper.py" 
 -file "/root/reducer.py"

注意：$HADOOP_HOME就是hadoop的家目录。
以上就是通过python完成词频统计的过程。