参考:http://www.nltk.org/
http://blog.csdn.net/tanzhangwen/article/details/8469491
分句:sent_tokenizer = nltk.data.load('tokenizer/punkt/english.pickle')
sentences = sent_tokenizer.tokenize(artcle)
分词:tokens = nltk.word_tokenize(sentence)
另一种分词的方法:from nltk.tokenize import WordPunctTokenizer
word_tokenizer = WordPunctTokenizer()
tokens = word_tokenizer.tokenzie(sentence)
词形还原:from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
word = lemmatizer.lemmatize('cats')
词性标注:tagged = nltk.pos_tag(tokens)
命名实体识别:entities = nltk.chunk.ne_chunk(tagged)
语法树:from nltk.corpus import treebank
t = treebank.parsed_sents(' wsj_001.mrg')[0]
t.draw()