Python自然语言处理--处理原始文本

NLTK API

导入工具包

>>from __future__ import division
>>import nltk,re,pprint

英文分词

>>raw = "I am a student......"
>>tokens = nltk.word_tokenize(raw)
# type(tokens)==list
# 分词列表创建nltk文本
>>text = nltk.Text(tokens)

从网页中提取文本

>>url = 'http://www.lfd.uci.edu/~gohlke/pythonlibs/#matplotlib'
>>html = urlopen(url).open()
>>raw = nltk.clean_html(html)
# clean_html()和clean_url()函数均已弃用，改用
# BeautifulSoup.get_text()对文本进行提取
>>from bs4 import BeautifulSoup
>>soup = BeautifulSoup(html)
>>raw = soup.get_text()
>>tokens = nltk.word_tokenize(raw)[10:21]
>>text = nltk.Text(tokens)
# 建立词汇表
>>words = [w.lower() for w in text]
>>vocab = sorted(set(words))