I am trying to print the top 10 frequent words using the following code. However, its not working. Any idea on how to fix it?
def reducer_count_words(self, word, counts):
# send all (num_occurrences, word) pairs to the same reducer.
# num_occurrences is so we can easily use Python's max() function.
yield None, (sum(counts), word)
# discard the key; it is just None
def reducer_find_max_10_words(self, _, word_count_pairs):
# each item of word_count_pairs is (count, word),
# so yielding one results in key=counts, value=word
tmp = sorted(word_count_pairs)[0:10]
yield tmp
解决方案tmp = sorted(word_count_pairs, key=lambda pair: pair[0], reverse=True)[0:10]
Explanation:
The key parameter of sorted() allows you to run a function on each element before comparison.
lambda pair: pair[0] is a function that extracts the number from your word_count_pairs.
reverse sorts in descending order, instead of ascending order.
Sources:
aside: If you have many different words, sorting the entire list to find the top ten is inefficient. There are much more efficient algorithms. The most_common() method mentioned in another answers probably utilizes a more efficient algorithm.