python清洗数据去除停用词_python – NLTK停用词删除问题

all_words = (w for w in all_words if w not in nltk.corpus.stopwords.words('english'))

它返回

Traceback (most recent call last):

File "fiction.py",line 8,in

我猜测停用词代码改变了用于’all_words’的对象类型,使得它们.key()函数无用.如何在使用键功能之前删除停用词而不更改其类型？完整代码如下：

import nltk

from nltk.corpus import PlaintextCorpusReader

corpus_root = './nltk_data/corpora/fiction'

fiction = PlaintextCorpusReader(corpus_root,'.*')

all_words=nltk.FreqDist(w.lower() for w in fiction.words())

all_words = (w for w in all_words if w not in nltk.corpus.stopwords.words('english'))

word_features = all_words.keys()[:100]

def document_features(document): # [_document-classify-extractor]

document_words = set(document) # [_document-classify-set]

features = {}

for word in word_features:

features['contains(%s)' % word] = (word in document_words)

return features

print document_features(fiction.words('fic/11.txt'))