我可以很容易地得到最常见的单词:stopwords = set(nltk.corpus.stopwords.words('english'))
tagged_words = nltk.word_tokenize(text)
tagged_words = nltk.pos_tag(tagged_words)
# Remove single-character tokens (mostly punctuation)
tagged_words = [tagged_word for tagged_word in tagged_words if len(tagged_word[0]) > 1]
# Remove numbers
tagged_words = [tagged_word for tagged_word in tagged_words if not tagged_word[0].isnumeric()]
# Remove stopwords
if remove_stopwords:
tagged_words = [tagged_word for tagged_word in tagged_words if tagged_word[0] not in stopwords]
# Dark magic
lemmatizer = nltk.stem.WordNetLemmatizer()
words = []
for tagged_word in tagged_words:
pos = wordnet_pos_code(tagged_word[1])
# Ignoring all words, except nouns, verbs, adjectives and adverbs
if pos is not None:
words.append({'word':lemmatizer.lemmatize(tagged_word[0], pos=pos), 'pos':tagged_word[1]})
# Calculate frequency distribution
fdist = nltk.FreqDist(words)
# Return top % words_count % words
res = []
for word, frequency in fdist.most_common(words_count):
word_dict = {}
word_dict['word'] = word
word_dict['count'] = frequency
res.append(word_dict)
return res
但我有一些词,比如“Brown”是人名,“Brown”是颜色,它们是不一样的。好的,我可以用大写字母核对。但如果我得到的是:Brown is not just a color. Brown is part of lifestyle. And Mr Brown should agree with me.
所以,nltk对词性分析做得很好。但如何才能得到最常见的词取决于词性?在