自然语言处理-chapter(1)--字符串操作

精通自然语言处理-chapter(1)–字符串操作

首先安装nltk包 sudo pip install nltk

安装完毕后 进入终端 下载nltk 数据源

 >>>  import nltk
 >>>  nltk.download()

1. 给定文本切分为语句

>>> import nltk
>>> from nltk.tokenize import sent_tokenize
>>> text='Welcome to my home'
>>> sent_tokenize(text)
['Welcome to my home']

1. 其他语言文本的切分

>>> import nltk
>>> french_tokenizer=nltk.data.load('tokenizers/punkt/french.pickle')
>>> french_tokenizer.tokenize('Deux agressions en quelques jours,voila ce qui a')
['Deux agressions en quelques jours,voila ce qui a']

1. 将句子切分为单词

>>> import nltk
>>> text=nltk.word_tokenize("Peter,59 years old, will join as a nonexecutive director on Nov. 29.>>")
>>> print(text)
['Peter,59', 'years', 'old', ',', 'will', 'join', 'as', 'a', 'nonexecutive', 'director', 'on', 'Nov.', '29', '.', '>', '>']   

分词器的继承树如下所示(部分):

1. 使用TreebankWordTokenizer 执行切分

>>> import nltk
>>> from nltk.tokenize import TreebankWordTokenizer
>>> tokenizer=TreebankWordTokenizer()
>>> tokenizer.tokenize("Have a nice day.I hope you find the book interesting")
['Have', 'a', 'nice', 'day.I', 'hope', 'you', 'find', 'the', 'book', 'interesting']

1. 使用正则表达式实现切分

>>> import nltk
>>> from nltk.tokenize import regexp_tokenize
>>> sent="Don't hesitate to ask questions"
>>> print(regexp_tokenize(sent,pattern='\w+|$[\d\.]+|\S+'))
['Don', "'t", 'hesitate', 'to', 'ask', 'questions']

 使用空格进行切分
>>> import nltk
>>> from nltk.tokenize import RegexpTokenizer
>>> tokenizer=RegexpTokenizer('\s',gaps=True)
>>> tokenizer.tokenize("Don't hesitate to ask questions")
["Don't", 'hesitate', 'to', 'ask', 'questions']

筛选大写字母开头的单词
>>> import nltk
>>> from nltk.tokenize import RegexpTokenizer
>>> sent="She is a girl.She is Good"
>>> capt=RegexpTokenizer('[A-Z]\w+')
>>> capt.tokenize(sent)
['She', 'She', 'Good']

标识符在语句中的位置和偏移量 使用nltk.tokenize.util模块
>>> import nltk
>>> from nltk.tokenize import WhitespaceTokenizer
>>> sent=' She is good\n'
>>> list(WhitespaceTokenizer().span_tokenize(sent))
[(1, 4), (5, 7), (8, 12)]

给定标识符序列 返回其跨度序列
>>> import nltk
>>> from nltk.tokenize import WhitespaceTokenizer
>>> from nltk.tokenize.util import spans_to_relative
>>> sent=" She is a good girl\n" 
>>> list(spans_to_relative(WhitespaceTokenizer().span_tokenize(sent)))
[(1, 3), (1, 2), (1, 1), (1, 4), (1, 4)]