WordCount Python版（整理）

突然间想起来，之前用过Python版的WordCount，之前没有做整理，现在想想还不晚，整理一下，说不定以后还会用到。

MapReduce我最近用的不多了，但是感觉不少业务场景，都可在WordCount 的基础上改进实现。

Python 具体实现（一个shell 脚本、一个Python脚本）:

##############################################################
# File Name: wordCount.sh
# Author: 
# mail: 
#=============================================================
#!/usr/bin/bash

cd `dirname $0`

source ~/.bashrc


in_dir=/user/lyx/input/*
out_dir=/user/lyx/output/

hadoop fs -rm -r -skipTrash $out_dir

hadoop jar $HADOOP_STREAMING \
				-D mapreduce.job.name="word count test" \
				-D mapreduce.job.queuename=root.xxxxxxxxxxxxx \
				-D mapred.map.tasks=500 \
				-D mapred.reduce.tasks=1000 \
				-input ${in_dir} \
				-output ${out_dir} \
				-file wordCount.py \
				-mapper "python wordCount.py mapper" \
				-reducer "python wordCount.py reducer"

##############################################################
# -*- coding=utf-8 -*-
# File Name: wordCount.py
# Author: 
# mail: 
# =============================================================
# !/usr/bin/python

from __future__ import absolute_import
from __future__ import print_function
from __future__ import division

import sys


def mapper():
    for line in sys.stdin:
        lsp = line.strip().split()
        for word in lsp:
            print(word + "\t" + str(1))


def reducer():
    current_key = None
    current_count = 0

    for line in sys.stdin:
        lsp = line.strip().split("\t")
        if len(lsp) < 2:
            continue

        key = lsp[0]
        count = int(lsp[1])

        if current_key == key:
            current_count += count
        else:
            if current_key:
                print(current_key + "\t" + str(current_count))
            current_key = key
            current_count = count
    print(current_key + "\t" + str(current_count))


if __name__ == "__main__":
    if sys.argv[1] == "mapper":
        mapper()
    elif sys.argv[1] == "reducer":
        reducer()

声明： 总结学习，有问题或不当之处，可以批评指正哦，谢谢。