python自然语言处理学习笔记一

第一章 语言处理与python

1 语言计算 文本与词汇

NLTK入门

下载安装nltk

http://www.nltk.org

下载数据

>>> import nltk

>>> nltk.download()

 

 下载完成,加载文本

>>> from nltk.book import *

*** Introductory Examples for the NLTK Book***

Loading text1, ..., text9 and sent1, ...,sent9

Type the name of the text or sentence toview it.

Type: 'texts()' or 'sents()' to list thematerials.

text1: Moby Dick by Herman Melville 1851

text2: Sense and Sensibility by Jane Austen1811

text3: The Book of Genesis

text4: Inaugural Address Corpus

text5: Chat Corpus

text6: Monty Python and the Holy Grail

text7: Wall Street Journal

text8: Personals Corpus

text9: The Man Who Was Thursday by G . K .Chesterton 1908

 

输入名字即可查询对应文本

>>> text1

<Text: Moby Dick by Herman Melville1851>

>>> text2

<Text: Sense and Sensibility by JaneAusten 1811>

 

搜索文本

>>> text1.concordance('monstrous')

Displaying 11 of 11 matches:

ong the former , one was of a mostmonstrous size . ... This came towards us ,

ON OF THE PSALMS . " Touching thatmonstrous bulk of the whale or ork we have r

ll over with a heathenish array ofmonstrous clubs and spears . Some were thick

d as you gazed , and wondered whatmonstrous cannibal and savage could ever hav

that has survived the flood ; mostmonstrous and most mountainous ! That Himmal

they might scout at Moby Dick as amonstrous fable , or still worse and more de

th of Radney .'" CHAPTER 55 Of theMonstrous Pictures of Whales . I shall ere l

ing Scenes . In connexion with themonstrous pictures of whales , I am strongly

ere to enter upon those still moremonstrous stories of them which are to be fo

ght have been rummaged out of thismonstrous cabinet there is no telling . But

of Whale - Bones ; for Whales of amonstrous size are oftentimes cast up dead u

 

#还有哪些词出现在相同的上文中

>>> text1.similar('monstrous')

impalpable puzzled part mystifying gamesomehorrible maddens

domineering curious exasperate untowardcontemptible careful

trustworthy delightfully christian meanuncommon abundant wise

>>> text2.similar('monstrous')

very exceedingly so heartily as vastamazingly extremely great sweet a

remarkably good

 

 

>>>text2.common_contexts(['monstrous','very'])

be_glad a_lucky am_glad is_pretty a_pretty

 

#美国总统就职演说词汇分布图

>>> text4.dispersion_plot(['citizens','democracy','freedom','duties','America'])

 

#产生随机文本

>>> text3.generate()

Traceback (most recent call last):

 File "<stdin>", line 1, in <module>

AttributeError: 'Text' object has noattribute 'generate'

 

#计数词汇

>>> len(text3)  #创世纪有44764个单词和标点符号

44764

 

 

#取得排序后的词汇条目

>>> sorted(set(text3))

['!', "'", '(', ')', ',', ',)','.', '.)', ':', ';', ';)', '?', '?)', 'A', 'Abel', 'Abelmizraim', 'Abidah','Abide', 'Abimael', 'Abimelech', 'Abr', 'A

brah', 'Abraham', 'Abram', 'Accad', 'Achbor','Adah', 'Adam', 'Adbeel', 'Admah', 'Adullamite', 'After', 'Aholibamah','Ahuzzath', 'Ajah', 'Akan', 'All

', 'Allonbachuth', 'Almighty', 'Almodad','Also', 'Alvah', 'Alvan', 'Am', 'Amal', 'Amalek', 'Amalekites', 'Ammon','Amorite', 'Amorites', 'Amraphel',