python tokenize_python – 滥用nltk的word_tokenize(已发送)的后果

nltk.tokenize.word_tokenize(text)只是一个瘦的

wrapper function,它调用

TreebankWordTokenizer类实例的tokenize方法,它显然使用简单的正则表达式来解析一个句子.

该类的文档声明：

This tokenizer assumes that the text has already been segmented into

sentences. Any periods — apart from those at the end of a string —

are assumed to be part of the word they are attached to (e.g. for

abbreviations,etc),and are not separately tokenized.

底层tokenize方法本身非常简单：

def tokenize(self,text):

for regexp in self.CONTRACTIONS2:

text = regexp.sub(r'\1 \2',text)

for regexp in self.CONTRACTIONS3:

text = regexp.sub(r'\1 \2 \3',text)

# Separate most punctuation

text = re.sub(r"([^\w\.\'\-\/,&])",r' \1 ',text)

# Separate commas if they're followed by space.

# (E.g.,don't separate 2,500)

text = re.sub(r"(,\s)",r' \1',text)

# Separate single quotes if they're followed by a space.

text = re.sub(r"('\s)",text)

# Separate periods that come before newline or end of string.

text = re.sub('\. *(\n|$)',' . ',text)

return text.split()

基本上,该方法通常做的是将句点标记为单独的标记,如果它落在字符串的末尾：

>>> nltk.tokenize.word_tokenize("Hello,world.")

['Hello',','world','.']

落在字符串中的任何句点都被标记为单词的一部分,假设它是缩写：

>>> nltk.tokenize.word_tokenize("Hello,world. How are you?")

['Hello','world.','How','are','you','?']

只要这种行为是可以接受的,你应该没事.