［MapReduce］Top 10 标签

源自Udacity，Intro to hadoop这门课Final Project

题目要求，从给定的论坛帖子数据中，找出被使用最多的10个tag。tag存在于question中（因为comment和answer也存在于数据表中，所有需要在mapper中过滤）

思路：

mapper 从是question的记录中，提取tags，并输出

reducer 给每个tags计数，保存到一个dict里面，最后排序输出 

# mapper
#!/usr/bin/python

import sys
import csv

reader = csv.reader(sys.stdin, delimiter = "\t")
next(reader, None)

for line in reader:
    if len(line) != 19:
        continue

    node_type = line[5].strip()
    if node_type == "question":
        tags = line[2].strip().split()
        for tag in tags:
            print tag

# reducer
#!/usr/bin/python

import sys
import csv

tags = {}
oldKey = None
count = 0

for line in sys.stdin:
    data = line.strip().split("\t")
    if len(data) != 1:
        continue

    thisKey = data[0]

    if oldKey and thisKey != oldKey:
        tags[oldKey] = count
        count = 0
        oldKey = thisKey

    oldKey = thisKey
    count += 1

if oldKey != None:
    tags[oldKey] = count

top10 = sorted(tags, key=tags.get, reverse=True)[:10]
for tag in top10:
    print "{0}\t{1}".format(tag, tags[tag])