python tokenizer_nltk RegexpTokenizer类:python自然语言处理

前面的一些分词工具都是写好的的规则

如果我们想按照自己的规则进行分词 可以使用正则分词器

1.RegexpTokenizer类

from nltk.tokenize import RegexpTokenizer

text = " I won't just survive, Oh, you will see me thrive. Can't write my story,I'm beyond the archetype."

# 实例化RegexpTokenizer 会按照正则表达式进行re.findall()

regexp_tokenizer = RegexpTokenizer(pattern="\w+")

# 实例化RegexpTokenizer 指定gaps=True会按照正则表达式进行re.split()

regexp_tokenizer1 = RegexpTokenizer("[\s,'\.]", gaps=True)

print(regexp_tokenizer.tokenize(text))

# ['I', 'won', 't', 'just', 'survive', 'Oh', 'you', 'will', 'see', 'me', 'thrive', 'Can', 't', 'write', 'my', 'story', 'I', 'm', 'beyond', 'the', 'archetype']

print(regexp_tokenizer1.tokenize(text))

# ['I', 'won', 't', 'just', 'survive', 'Oh', 'you', 'will', 'see', 'me', 'thrive', 'Can', 't', 'write', 'my', 'story', 'I', 'm', 'beyond', 'the', 'archetype']

---------------------

作者：qq_41864652