python处理词项的停用词_如何使用Python nltk.tokenize将包含停用词的短语作为单个标记对待...

您可以使用nltk的Multi-Word Expression Tokenizer,它可以将多单词表达式合并为单个标记.您可以创建一个包含多词表达式的词典,并向其添加条目,如下所示：

from nltk.tokenize import MWETokenizer

mwetokenizer = MWETokenizer([('President','of','the','United','States')],separator=' ')

mwetokenizer.add_mwe(('President','France'))

请注意,MWETokenizer将带标记文本的列表作为输入,然后对其进行重新标记.因此,首先标记该句子.使用word_tokenize(),然后将其输入MWETokenizer：

from nltk.tokenize import word_tokenize

sentence = "Trump is the President of the United States,and Macron is the President of France."

mwetokenized_sentence = mwetokenizer.tokenize(word_tokenize(sentence))

# ['Trump','is','President of the United States',','and','Macron','President of France','.']

然后,过滤掉停用词以获得最终过滤的标记化句子：

from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))

filtered_sentence = [token for token in mwetokenizer.tokenize(word_tokenize(sentence)) if token not in stop_words]

print(filtered_sentence)

输出：

['Trump','.']