java正则表达式 分词_自然语言处理3.7——用正则表达式为文本分词

1、分词的简单方法：

在空格字符处分割文本是文本分词最简单的方法。考虑一下摘自《爱丽丝梦游仙境》中的文本。

>>> raw = """'When I'M a Duchess,' she said to herself, (not in a very hopeful tone

... though), 'I won't have any pepper in my kitchen AT ALL. Soup does very

... well without--Maybe it's always pepper that makes people hot-tempered,'..."""

可以使用raw.split()在空格符处分割原始文本。使用正则表达式能做同样的事情，匹配字符串中的所有空白符是远远不够的，因为这会导致结果中包含'\n'换行符。需要同时匹配任何数量的空格符、制表符或者换行符。

>>>import re

>>>re.split(r' ',raw)

["'When", "I'M", 'a', "Duchess,'", 'she', 'said', 'to', 'herself,', '(not', 'in',

'a', 'very', 'hopeful', 'tone\nthough),', "'I", "won't", 'have', 'any', 'pepper',

'in', 'my', 'kitchen', 'AT', 'ALL.', 'Soup', 'does', 'very\nwell', 'without--Maybe',

"it's", 'always', 'pepper', 'that', 'makes', 'people', "hot-tempered,'..."]

>>>re.split(r'