Python利用hadoop Streaming编写的Map-Reduce程序命令运行和本地调试运行

Python利用hadoop Streaming编写的Map-Reduce程序命令运行和本地调试运行

一、Hadoop的 Streaming介绍

     Hadoop本身是用Java开发的，程序也需要用Java编写，但是通过Hadoop Streaming，可以使 用任意语言（python、ruby、C++等）来编写程序，让Hadoop运行。

       Hadoop提供了MapReduce的API,并允许使用非java的其他语言来写自己的map和reduce函数。而Streaming使用Unix标准流作为Hadoop和应用程序之间的接口，所以允许我们可以使用任何编程语言通过标准输入/输出来写MapReduce程序。Streaming的相关源码可以在hadoop的源码中查看。

    Streaming天生适合用于文本处理，在文本模式下使用时，它有一个数据的行视图。map的输入数据通过标准输入流传递给map函数，并且是一行一行地传输，最后结果写入标准输出。map输出的键值对以一个制表符分割的行，它以这样的形式写到标准输出。reduce函数的输入格式相同——通过制表符来分割的键值对——并通过标准输入流进行传输。reduce函数从标准输入流中读取输入行，该输入已由hadoop框架根据键排序，最后将结果写入标准输出。

二、评价Hadoop的 Streaming

• 优点
  • 可以使用自己喜欢的语言来编写MapReduce程序 
    
  • 不需要像写Java的MR程序那样import一大堆库，在代码里做一大堆配置，很多东西都抽象到了stdio上，代码量显著减少
  • 因为没有库的依赖，调试方便，并且可以脱离Hadoop先在本地用管道pipes模拟调试
• 缺点
  • 只能通过命令行参数来控制MapReduce框架，不像Java的程序那样可以在代码里使用API，控制力比较弱，有些东西鞭长莫及
  • 因为中间隔着一层处理，效率会比较慢

所以Hadoop Streaming比较适合做一些简单的任务，比如用python写只有一两百行的脚本。如果项目比较复杂，或者需要进行比较细致的优化，使用Streaming就容易出现一些束手束脚的地方。

三、Python编写的MapReduce程序

1） 输入的数据为books.json：   把文本数据放入fs建立好的input文件夹中

["milton-paradise.txt", "[ Paradise Lost by John Milton 1667 ] Book I Of Man ' s first disobedience , and the fruit Of that forbidden tree whose mortal taste Brought death into the World , and all our woe , With loss of Eden , till one greater Man Restore us , and regain the blissful seat , Sing , Heavenly Muse , that , on the secret top Of Oreb , or of Sinai , didst inspire That shepherd who first taught the chosen seed In the beginning how the heavens and earth Rose out of Chaos : or , if Sion hill Delight thee more , and Siloa ' s brook that flowed Fast by the oracle of God , I thence Invoke thy aid to my adventurous song , That with no middle flight intends to soar Above th ' Aonian mount , while it pursues Things unattempted yet in prose or rhyme ."]
["edgeworth-parents.txt", "[ The Parent ' s Assistant , by Maria Edgeworth ] THE ORPHANS . Near the ruins of the castle of Rossmore , in Ireland , is a small cabin , in which there once lived a widow and her four children . As long as she was able to work , she was very industrious , and was accounted the best spinner in the parish ; but she overworked herself at last , and fell ill , so that she could not sit to her wheel as she used to do , and was obliged to give it up to her eldest daughter , Mary ."]
["austen-emma.txt", "[ Emma by Jane Austen 1816 ] VOLUME I CHAPTER I Emma Woodhouse , handsome , clever , and rich , with a comfortable home and happy disposition , seemed to unite some of the best blessings of existence ; and had lived nearly twenty - one years in the world with very little to distress or vex her . She was the youngest of the two daughters of a most affectionate , indulgent father ; and had , in consequence of her sister ' s marriage , been mistress of his house from a very early period . Her mother had died too long ago for her to have more than an indistinct remembrance of her caresses ; and her place had been supplied by an excellent woman as governess , who had fallen little short of a mother in affection ."]
["chesterton-ball.txt", "[ The Ball and The Cross by G . K . Chesterton 1909 ] I . A DISCUSSION SOMEWHAT IN THE AIR The flying ship of Professor Lucifer sang through the skies like a silver arrow ; the bleak white steel of it , gleaming in the bleak blue emptiness of the evening . That it was far above the earth was no expression for it ; to the two men in it , it seemed to be far above the stars . The professor had himself invented the flying machine , and had also invented nearly everything in it ."]
["bible-kjv.txt", "[ The King James Bible ] The Old Testament of the King James Bible The First Book of Moses : Called Genesis 1 : 1 In the beginning God created the heaven and the earth . 1 : 2 And the earth was without form , and void ; and darkness was upon the face of the deep . And the Spirit of God moved upon the face of the waters . 1 : 3 And God said , Let there be light : and there was light . 1 : 4 And God saw the light , that it was good : and God divided the light from the darkness . 1 : 5 And God called the light Day , and the darkness he called Night . And the evening and the morning were the first day ."]
["chesterton-thursday.txt", "[ The Man Who Was Thursday by G . K . Chesterton 1908 ] To Edmund Clerihew Bentley A cloud was on the mind of men , and wailing went the weather , Yea , a sick cloud upon the soul when we were boys together . Science announced nonentity and art admired decay ; The world was old and ended : but you and I were gay ; Round us in antic order their crippled vices came -- Lust that had lost its laughter , fear that had lost its shame . Like the white lock of Whistler , that lit our aimless gloom , Men showed their own white feather as proudly as a plume . Life was a fly that faded , and death a drone that stung ; The world was very old indeed when you and I were young ."]
["blake-poems.txt", "[ Poems by William Blake 1789 ] SONGS OF INNOCENCE AND OF EXPERIENCE and THE BOOK of THEL SONGS OF INNOCENCE INTRODUCTION Piping down the valleys wild , Piping songs of pleasant glee , On a cloud I saw a child , And he laughing said to me : \" Pipe a song about a Lamb !\" So I piped with merry cheer . \" Piper , pipe that song again ;\" So I piped : he wept to hear . \" Drop thy pipe , thy happy pipe ; Sing thy songs of happy cheer :!\" So I sang the same again , While he wept with joy to hear . \" Piper , sit thee down and write In a book , that all may read .\" So he vanish ' d from my sight ; And I pluck ' d a hollow reed , And I made a rural pen , And I stain ' d the water clear , And I wrote my happy songs Every child may joy to hear ."]
["shakespeare-caesar.txt", "[ The Tragedie of Julius Caesar by William Shakespeare 1599 ] Actus Primus . Scoena Prima . Enter Flauius , Murellus , and certaine Commoners ouer the Stage . Flauius . Hence : home you idle Creatures , get you home : Is this a Holiday ? What , know you not ( Being Mechanicall ) you ought not walke Vpon a labouring day , without the signe Of your Profession ? Speake , what Trade art thou ? Car . Why Sir , a Carpenter Mur . Where is thy Leather Apron , and thy Rule ? What dost thou with thy best Apparrell on ? You sir , what Trade are you ? Cobl . Truely Sir , in respect of a fine Workman , I am but as you would say , a Cobler Mur . But what Trade art thou ? Answer me directly Cob . A Trade Sir , that I hope I may vse , with a safe Conscience , which is indeed Sir , a Mender of bad soules Fla ."]
["whitman-leaves.txt", "[ Leaves of Grass by Walt Whitman 1855 ] Come , said my soul , Such verses for my Body let us write , ( for we are one ,) That should I after return , Or , long , long hence , in other spheres , There to some group of mates the chants resuming , ( Tallying Earth ' s soil , trees , winds , tumultuous waves ,) Ever with pleas ' d smile I may keep on , Ever and ever yet the verses owning -- as , first , I here and now Signing for Soul and Body , set to them my name , Walt Whitman [ BOOK I . INSCRIPTIONS ] } One ' s - Self I Sing One ' s - self I sing , a simple separate person , Yet utter the word Democratic , the word En - Masse ."]
["melville-moby_dick.txt", "[ Moby Dick by Herman Melville 1851 ] ETYMOLOGY . ( Supplied by a Late Consumptive Usher to a Grammar School ) The pale Usher -- threadbare in coat , heart , body , and brain ; I see him now . He was ever dusting his old lexicons and grammars , with a queer handkerchief , mockingly embellished with all the gay flags of all the known nations of the world . He loved to dust his old grammars ; it somehow mildly reminded him of his mortality ."]

2） 把文本数据books.json放入fs建立好的1024_input文件夹中

localhost:experiment_data a6$ hadoop fs -mkdir /1024_input
localhost:experiment_data a6$ hadoop dfs -ls /
localhost:experiment_data a6$ hadoop fs -copyFromLocal book.json /1024_input
上传本地到hdfs上的命令格式如下：
$ hadoop fs -copyFromLocal <PATH TO LOCAL FILE(S)> input/

3） 编写代码之一：统计每个单词在哪些文档里出现

map函数 invertedMapper.py 文件，内容如下：

#!/usr/bin/env python
'''
Test by yyz
Date: 2017/10/24
'''
import sys
import json
for line in sys.stdin:
	line = line.strip()
	record = json.loads(line)
	fileName = record[0];
	value = record[1];
	words = value.split()
	for word in words:
		print '%s\t%s' % (word, fileName)

4） 编写代码之二：形成单词的简单倒排索引

reduce函数 invertedReducer.py 文件，内容如下:

# !/usr/bin/env python
'''
Test by yyz
Date: 2017/10/24
'''
import sys
# maps words to their counts
media = {}
wordInMedia = {}
# input comes from STDIN
for line in sys.stdin:
    (word, fileName) = line.strip().split('\t', 1)
    media.setdefault(word, [])
    media[word].append(fileName)
for word in media:
    wordInMedia.setdefault(word, list(set(media[word])))
# write the results to STDOUT (standard output)
for word in wordInMedia:
    print '%s\t%s' % (word, wordInMedia[word])

四、运行Python编写的MapReduce程序

在hadoop-2.6.5中测试本代码

1）确保输出文件夹1024_output不存在

localhost:experiment_data a6$ hadoop fs -rm -r /1024_output

2）在本地进行测试，书写模式如下：

cat <input path> | python <path to mapper script> | sort -t $'\t' -k1,1 | python <path to reducer script> > <output path>

我的本地测试命令如下：

cat map_reduce_program/book.json | python map_reduce_program/invertedMapper.py | sort | python map_reduce_program/invertedReducer.py

2）hadoop命令运行MR程序，书写模式如下：

$hadoop jar /Users/a6/Applications/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar \
-mapper 'python mapper.py'  
-file mapper.py\
-reducer 'python reducer.py' 
-file reducer.py \
-input input/* 
-output output

我的运行命令如下：

localhost:~ a6$ hadoop jar /Users/a6/Applications/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar  -mapper "python invertedMapper.py" -reducer "python invertedReducer.py" -input "/1024_input/*" -output "/1024_output" -file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedMapper.py" -file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedReducer.py"

书写格式如下：

localhost:~ a6$ hadoop jar /Users/a6/Applications/hadoop-2.6.5/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar  \
-mapper "python invertedMapper.py" \
-reducer "python invertedReducer.py" \
-input "/1024_input/*"  \
-output "/1024_output" \
-file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedMapper.py"  \
-file "/Users/a6/Downloads/PycharmProjects/map_reduce_program/invertedReducer.py"

五、注意事项

 1）Hadoop默认按照tab（\t）来分割key和value，以第一个分割出的部分为key，按key进行排序，因此这里使用
    sort -t $'\t' -k1,1    来模拟。
 2）如果你有其他需求，在交给Hadoop Streaming执行时可以通过命令行参数调，本地调试也可以进行相应的调整，主要是调整sort的参数。因此为了能够熟练进行本地调试，建议先掌握sort命令的用法。
 3）如果你在python脚本里加上了shabang标记（#! /usr/bin/env python），并且为它们添加了执行权限，也可以用类似于
    ./mapper.py     代替     python mapper.py

六、生成结果如下

all	['milton-paradise.txt', 'blake-poems.txt', 'melville-moby_dick.txt']
Rossmore	['edgeworth-parents.txt']
Consumptive	['melville-moby_dick.txt']
Self	['whitman-leaves.txt']
Night	['bible-kjv.txt']
eldest	['edgeworth-parents.txt']
four	['edgeworth-parents.txt']
Caesar	['shakespeare-caesar.txt']
winds	['whitman-leaves.txt']
Moses	['bible-kjv.txt']
children	['edgeworth-parents.txt']
seemed	['chesterton-ball.txt', 'austen-emma.txt']
whose	['milton-paradise.txt']
Parent	['edgeworth-parents.txt']
father	['austen-emma.txt']
young	['chesterton-thursday.txt']
vex	['austen-emma.txt']
to	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'blake-poems.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
th	['milton-paradise.txt']
smile	['whitman-leaves.txt']
Day	['bible-kjv.txt']
hill	['milton-paradise.txt']
me	['blake-poems.txt', 'shakespeare-caesar.txt']
woman	['austen-emma.txt']
Whitman	['whitman-leaves.txt']
song	['milton-paradise.txt', 'blake-poems.txt']
far	['chesterton-ball.txt']
void	['bible-kjv.txt']
rhyme	['milton-paradise.txt']
Oreb	['milton-paradise.txt']
Pipe	['blake-poems.txt']
Where	['shakespeare-caesar.txt']
heaven	['bible-kjv.txt']
Bentley	['chesterton-thursday.txt']
--	['whitman-leaves.txt', 'melville-moby_dick.txt', 'chesterton-thursday.txt']
sang	['chesterton-ball.txt', 'blake-poems.txt']
1667	['milton-paradise.txt']
pleas	['whitman-leaves.txt']
Paradise	['milton-paradise.txt']
Fast	['milton-paradise.txt']
VOLUME	['austen-emma.txt']
Shakespeare	['shakespeare-caesar.txt']
earth	['chesterton-ball.txt', 'bible-kjv.txt', 'milton-paradise.txt']
Jane	['austen-emma.txt']
emptiness	['chesterton-ball.txt']
blue	['chesterton-ball.txt']
what	['shakespeare-caesar.txt']
darkness	['bible-kjv.txt']
Cobler	['shakespeare-caesar.txt']
supplied	['austen-emma.txt']
Delight	['milton-paradise.txt']
above	['chesterton-ball.txt']
;	['austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
ever	['whitman-leaves.txt', 'melville-moby_dick.txt']
piped	['blake-poems.txt']
Apparrell	['shakespeare-caesar.txt']
body	['melville-moby_dick.txt']
Emma	['austen-emma.txt']
Professor	['chesterton-ball.txt']
men	['chesterton-ball.txt', 'chesterton-thursday.txt']
here	['whitman-leaves.txt']
water	['blake-poems.txt']
let	['whitman-leaves.txt']
crippled	['chesterton-thursday.txt']
sing	['whitman-leaves.txt']
[	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
invented	['chesterton-ball.txt']
daughter	['edgeworth-parents.txt']
my	['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt']
spheres	['whitman-leaves.txt']
DISCUSSION	['chesterton-ball.txt']
Blake	['blake-poems.txt']
heart	['melville-moby_dick.txt']
Things	['milton-paradise.txt']
Actus	['shakespeare-caesar.txt']
till	['milton-paradise.txt']
Book	['milton-paradise.txt', 'bible-kjv.txt']
1908	['chesterton-thursday.txt']
1909	['chesterton-ball.txt']
Answer	['shakespeare-caesar.txt']
cloud	['blake-poems.txt', 'chesterton-thursday.txt']
from	['bible-kjv.txt', 'austen-emma.txt', 'blake-poems.txt']
would	['shakespeare-caesar.txt']
give	['edgeworth-parents.txt']
two	['chesterton-ball.txt', 'austen-emma.txt']
more	['milton-paradise.txt', 'austen-emma.txt']
Piper	['blake-poems.txt']
Leather	['shakespeare-caesar.txt']
about	['blake-poems.txt']
Rule	['shakespeare-caesar.txt']
So	['blake-poems.txt']
excellent	['austen-emma.txt']
known	['melville-moby_dick.txt']
fly	['chesterton-thursday.txt']
Signing	['whitman-leaves.txt']
word	['whitman-leaves.txt']
middle	['milton-paradise.txt']
this	['shakespeare-caesar.txt']
work	['edgeworth-parents.txt']
soul	['whitman-leaves.txt', 'chesterton-thursday.txt']
hope	['shakespeare-caesar.txt']
INNOCENCE	['blake-poems.txt']
laughter	['chesterton-thursday.txt']
admired	['chesterton-thursday.txt']
shepherd	['milton-paradise.txt']
lock	['chesterton-thursday.txt']
Edmund	['chesterton-thursday.txt']
Aonian	['milton-paradise.txt']
sir	['shakespeare-caesar.txt']
sit	['blake-poems.txt', 'edgeworth-parents.txt']
divided	['bible-kjv.txt']
Who	['chesterton-thursday.txt']
Carpenter	['shakespeare-caesar.txt']
,)	['whitman-leaves.txt']
machine	['chesterton-ball.txt']
how	['milton-paradise.txt']
Truely	['shakespeare-caesar.txt']
Why	['shakespeare-caesar.txt']
embellished	['melville-moby_dick.txt']
A	['chesterton-ball.txt', 'shakespeare-caesar.txt', 'chesterton-thursday.txt']
may	['blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt']
ORPHANS	['edgeworth-parents.txt']
after	['whitman-leaves.txt']
:!"	['blake-poems.txt']
consequence	['austen-emma.txt']
waves	['whitman-leaves.txt']
Hence	['shakespeare-caesar.txt']
bleak	['chesterton-ball.txt']
a	['austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
short	['austen-emma.txt']
coat	['melville-moby_dick.txt']
One	['whitman-leaves.txt']
so	['edgeworth-parents.txt']
order	['chesterton-thursday.txt']
signe	['shakespeare-caesar.txt']
What	['shakespeare-caesar.txt']
indeed	['shakespeare-caesar.txt', 'chesterton-thursday.txt']
Supplied	['melville-moby_dick.txt']
years	['austen-emma.txt']
ended	['chesterton-thursday.txt']
!"	['blake-poems.txt']
through	['chesterton-ball.txt']
same	['blake-poems.txt']
existence	['austen-emma.txt']
its	['chesterton-thursday.txt']
Body	['whitman-leaves.txt']
1	['bible-kjv.txt']
industrious	['edgeworth-parents.txt']
group	['whitman-leaves.txt']
decay	['chesterton-thursday.txt']
chosen	['milton-paradise.txt']
glee	['blake-poems.txt']
brook	['milton-paradise.txt']
,	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Sir	['shakespeare-caesar.txt']
Sing	['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt']
Tallying	['whitman-leaves.txt']
them	['whitman-leaves.txt']
evening	['chesterton-ball.txt', 'bible-kjv.txt']
Grammar	['melville-moby_dick.txt']
greater	['milton-paradise.txt']
died	['austen-emma.txt']
Car	['shakespeare-caesar.txt']
thee	['milton-paradise.txt', 'blake-poems.txt']
safe	['shakespeare-caesar.txt']
Murellus	['shakespeare-caesar.txt']
She	['austen-emma.txt']
not	['shakespeare-caesar.txt', 'edgeworth-parents.txt']
now	['whitman-leaves.txt', 'melville-moby_dick.txt']
silver	['chesterton-ball.txt']
gloom	['chesterton-thursday.txt']
name	['whitman-leaves.txt']
arrow	['chesterton-ball.txt']
went	['chesterton-thursday.txt']
CHAPTER	['austen-emma.txt']
Mur	['shakespeare-caesar.txt']
Rose	['milton-paradise.txt']
taught	['milton-paradise.txt']
our	['milton-paradise.txt', 'chesterton-thursday.txt']
out	['milton-paradise.txt']
INSCRIPTIONS	['whitman-leaves.txt']
'	['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
day	['bible-kjv.txt', 'shakespeare-caesar.txt']
Muse	['milton-paradise.txt']
Genesis	['bible-kjv.txt']
William	['blake-poems.txt', 'shakespeare-caesar.txt']
flowed	['milton-paradise.txt']
waters	['bible-kjv.txt']
announced	['chesterton-thursday.txt']
Stage	['shakespeare-caesar.txt']
disobedience	['milton-paradise.txt']
G	['chesterton-ball.txt', 'chesterton-thursday.txt']
pleasant	['blake-poems.txt']
small	['edgeworth-parents.txt']
ETYMOLOGY	['melville-moby_dick.txt']
disposition	['austen-emma.txt']
Mechanicall	['shakespeare-caesar.txt']
mistress	['austen-emma.txt']
beginning	['milton-paradise.txt', 'bible-kjv.txt']
En	['whitman-leaves.txt']
created	['bible-kjv.txt']
Being	['shakespeare-caesar.txt']
could	['edgeworth-parents.txt']
keep	['whitman-leaves.txt']
fruit	['milton-paradise.txt']
place	['austen-emma.txt']
mortal	['milton-paradise.txt']
hence	['whitman-leaves.txt']
castle	['edgeworth-parents.txt']
fallen	['austen-emma.txt']
first	['milton-paradise.txt', 'bible-kjv.txt', 'whitman-leaves.txt']
Lamb	['blake-poems.txt']
flying	['chesterton-ball.txt']
There	['whitman-leaves.txt']
Speake	['shakespeare-caesar.txt']
one	['milton-paradise.txt', 'whitman-leaves.txt', 'austen-emma.txt']
Scoena	['shakespeare-caesar.txt']
directly	['shakespeare-caesar.txt']
hear	['blake-poems.txt']
Ever	['whitman-leaves.txt']
Sinai	['milton-paradise.txt']
owning	['whitman-leaves.txt']
Chesterton	['chesterton-ball.txt', 'chesterton-thursday.txt']
little	['austen-emma.txt']
"	['blake-poems.txt']
indulgent	['austen-emma.txt']
twenty	['austen-emma.txt']
top	['milton-paradise.txt']
their	['chesterton-thursday.txt']
2	['bible-kjv.txt']
too	['austen-emma.txt']
Such	['whitman-leaves.txt']
white	['chesterton-ball.txt', 'chesterton-thursday.txt']
cabin	['edgeworth-parents.txt']
that	['milton-paradise.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'edgeworth-parents.txt']
1599	['shakespeare-caesar.txt']
aimless	['chesterton-thursday.txt']
SOMEWHAT	['chesterton-ball.txt']
Democratic	['whitman-leaves.txt']
herself	['edgeworth-parents.txt']
than	['austen-emma.txt']
steel	['chesterton-ball.txt']
loved	['melville-moby_dick.txt']
showed	['chesterton-thursday.txt']
Come	['whitman-leaves.txt']
tree	['milton-paradise.txt']
nations	['melville-moby_dick.txt']
idle	['shakespeare-caesar.txt']
Usher	['melville-moby_dick.txt']
light	['bible-kjv.txt']
threadbare	['melville-moby_dick.txt']
indistinct	['austen-emma.txt']
and	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
handkerchief	['melville-moby_dick.txt']
God	['milton-paradise.txt', 'bible-kjv.txt']
labouring	['shakespeare-caesar.txt']
mind	['chesterton-thursday.txt']
comfortable	['austen-emma.txt']
Late	['melville-moby_dick.txt']
say	['shakespeare-caesar.txt']
mates	['whitman-leaves.txt']
have	['austen-emma.txt']
blissful	['milton-paradise.txt']
saw	['bible-kjv.txt', 'blake-poems.txt']
Invoke	['milton-paradise.txt']
self	['whitman-leaves.txt']
able	['edgeworth-parents.txt']
soules	['shakespeare-caesar.txt']
lit	['chesterton-thursday.txt']
also	['chesterton-ball.txt']
forbidden	['milton-paradise.txt']
drone	['chesterton-thursday.txt']
which	['shakespeare-caesar.txt', 'edgeworth-parents.txt']
flags	['melville-moby_dick.txt']
With	['milton-paradise.txt']
Spirit	['bible-kjv.txt']
pursues	['milton-paradise.txt']
soar	['milton-paradise.txt']
Drop	['blake-poems.txt']
most	['austen-emma.txt']
Piping	['blake-poems.txt']
Enter	['shakespeare-caesar.txt']
The	['chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'shakespeare-caesar.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
]	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Called	['bible-kjv.txt']
SONGS	['blake-poems.txt']
Restore	['milton-paradise.txt']
clear	['blake-poems.txt']
face	['bible-kjv.txt']
pipe	['blake-poems.txt']
accounted	['edgeworth-parents.txt']
Earth	['whitman-leaves.txt']
}	['whitman-leaves.txt']
came	['chesterton-thursday.txt']
inspire	['milton-paradise.txt']
wailing	['chesterton-thursday.txt']
queer	['melville-moby_dick.txt']
utter	['whitman-leaves.txt']
fear	['chesterton-thursday.txt']
heavens	['milton-paradise.txt']
fine	['shakespeare-caesar.txt']
(	['shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt']
plume	['chesterton-thursday.txt']
should	['whitman-leaves.txt']
rich	['austen-emma.txt']
stain	['blake-poems.txt']
handsome	['austen-emma.txt']
mortality	['melville-moby_dick.txt']
do	['edgeworth-parents.txt']
his	['melville-moby_dick.txt', 'austen-emma.txt']
Prima	['shakespeare-caesar.txt']
get	['shakespeare-caesar.txt']
Leaves	['whitman-leaves.txt']
trees	['whitman-leaves.txt']
While	['blake-poems.txt']
nearly	['chesterton-ball.txt', 'austen-emma.txt']
THE	['chesterton-ball.txt', 'blake-poems.txt', 'edgeworth-parents.txt']
him	['melville-moby_dick.txt']
child	['blake-poems.txt']
Maria	['edgeworth-parents.txt']
brain	['melville-moby_dick.txt']
EXPERIENCE	['blake-poems.txt']
morning	['bible-kjv.txt']
overworked	['edgeworth-parents.txt']
bad	['shakespeare-caesar.txt']
antic	['chesterton-thursday.txt']
she	['edgeworth-parents.txt']
Austen	['austen-emma.txt']
ruins	['edgeworth-parents.txt']
wrote	['blake-poems.txt']
widow	['edgeworth-parents.txt']
set	['whitman-leaves.txt']
art	['shakespeare-caesar.txt', 'chesterton-thursday.txt']
Chaos	['milton-paradise.txt']
seat	['milton-paradise.txt']
ouer	['shakespeare-caesar.txt']
see	['melville-moby_dick.txt']
Lucifer	['chesterton-ball.txt']
are	['shakespeare-caesar.txt', 'whitman-leaves.txt']
John	['milton-paradise.txt']
best	['shakespeare-caesar.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
said	['bible-kjv.txt', 'whitman-leaves.txt', 'blake-poems.txt']
That	['chesterton-ball.txt', 'milton-paradise.txt', 'whitman-leaves.txt']
INTRODUCTION	['blake-poems.txt']
3	['bible-kjv.txt']
woe	['milton-paradise.txt']
marriage	['austen-emma.txt']
K	['chesterton-ball.txt', 'chesterton-thursday.txt']
King	['bible-kjv.txt']
we	['whitman-leaves.txt', 'chesterton-thursday.txt']
Bible	['bible-kjv.txt']
Commoners	['shakespeare-caesar.txt']
joy	['blake-poems.txt']
were	['bible-kjv.txt', 'chesterton-thursday.txt']
pluck	['blake-poems.txt']
merry	['blake-poems.txt']
AND	['blake-poems.txt']
youngest	['austen-emma.txt']
last	['edgeworth-parents.txt']
Yea	['chesterton-thursday.txt']
ill	['edgeworth-parents.txt']
thence	['milton-paradise.txt']
called	['bible-kjv.txt']
s	['milton-paradise.txt', 'whitman-leaves.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
mildly	['melville-moby_dick.txt']
Sion	['milton-paradise.txt']
expression	['chesterton-ball.txt']
Yet	['whitman-leaves.txt']
Conscience	['shakespeare-caesar.txt']
THEL	['blake-poems.txt']
distress	['austen-emma.txt']
Her	['austen-emma.txt']
Clerihew	['chesterton-thursday.txt']
simple	['whitman-leaves.txt']
period	['austen-emma.txt']
hollow	['blake-poems.txt']
gleaming	['chesterton-ball.txt']
Ireland	['edgeworth-parents.txt']
Cobl	['shakespeare-caesar.txt']
respect	['shakespeare-caesar.txt']
Holiday	['shakespeare-caesar.txt']
Was	['chesterton-thursday.txt']
Flauius	['shakespeare-caesar.txt']
Apron	['shakespeare-caesar.txt']
Mender	['shakespeare-caesar.txt']
dusting	['melville-moby_dick.txt']
been	['austen-emma.txt']
.	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
secret	['milton-paradise.txt']
feather	['chesterton-thursday.txt']
affectionate	['austen-emma.txt']
lived	['edgeworth-parents.txt', 'austen-emma.txt']
flight	['milton-paradise.txt']
gay	['melville-moby_dick.txt', 'chesterton-thursday.txt']
am	['shakespeare-caesar.txt']
deep	['bible-kjv.txt']
World	['milton-paradise.txt']
Whistler	['chesterton-thursday.txt']
Round	['chesterton-thursday.txt']
And	['bible-kjv.txt', 'blake-poems.txt']
Old	['bible-kjv.txt']
mount	['milton-paradise.txt']
thou	['shakespeare-caesar.txt']
while	['milton-paradise.txt']
seed	['milton-paradise.txt']
ought	['shakespeare-caesar.txt']
lexicons	['melville-moby_dick.txt']
wild	['blake-poems.txt']
aid	['milton-paradise.txt']
Creatures	['shakespeare-caesar.txt']
soil	['whitman-leaves.txt']
is	['shakespeare-caesar.txt', 'edgeworth-parents.txt']
Dick	['melville-moby_dick.txt']
it	['chesterton-ball.txt', 'bible-kjv.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'edgeworth-parents.txt']
AIR	['chesterton-ball.txt']
shame	['chesterton-thursday.txt']
grammars	['melville-moby_dick.txt']
good	['bible-kjv.txt']
Siloa	['milton-paradise.txt']
in	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
prose	['milton-paradise.txt']
return	['whitman-leaves.txt']
You	['shakespeare-caesar.txt']
vanish	['blake-poems.txt']
if	['milton-paradise.txt']
1816	['austen-emma.txt']
)	['shakespeare-caesar.txt', 'melville-moby_dick.txt']
parish	['edgeworth-parents.txt']
reminded	['melville-moby_dick.txt']
unite	['austen-emma.txt']
chants	['whitman-leaves.txt']
Of	['milton-paradise.txt', 'shakespeare-caesar.txt']
Workman	['shakespeare-caesar.txt']
daughters	['austen-emma.txt']
wheel	['edgeworth-parents.txt']
used	['edgeworth-parents.txt']
I	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt']
upon	['bible-kjv.txt', 'chesterton-thursday.txt']
Soul	['whitman-leaves.txt']
Cob	['shakespeare-caesar.txt']
IN	['chesterton-ball.txt']
dust	['melville-moby_dick.txt']
thy	['milton-paradise.txt', 'blake-poems.txt', 'shakespeare-caesar.txt']
Is	['shakespeare-caesar.txt']
didst	['milton-paradise.txt']
nonentity	['chesterton-thursday.txt']
person	['whitman-leaves.txt']
cheer	['blake-poems.txt']
without	['bible-kjv.txt', 'shakespeare-caesar.txt']
resuming	['whitman-leaves.txt']
In	['milton-paradise.txt', 'bible-kjv.txt', 'blake-poems.txt']
very	['edgeworth-parents.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
the	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
1789	['blake-poems.txt']
-	['whitman-leaves.txt', 'austen-emma.txt']
verses	['whitman-leaves.txt']
Tragedie	['shakespeare-caesar.txt']
Heavenly	['milton-paradise.txt']
yet	['milton-paradise.txt', 'whitman-leaves.txt']
dost	['shakespeare-caesar.txt']
vices	['chesterton-thursday.txt']
death	['milton-paradise.txt', 'chesterton-thursday.txt']
Lust	['chesterton-thursday.txt']
mother	['austen-emma.txt']
had	['chesterton-ball.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
intends	['milton-paradise.txt']
adventurous	['milton-paradise.txt']
spinner	['edgeworth-parents.txt']
book	['blake-poems.txt']
4	['bible-kjv.txt']
skies	['chesterton-ball.txt']
On	['blake-poems.txt']
certaine	['shakespeare-caesar.txt']
read	['blake-poems.txt']
James	['bible-kjv.txt']
stung	['chesterton-thursday.txt']
early	['austen-emma.txt']
know	['shakespeare-caesar.txt']
world	['melville-moby_dick.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
Grass	['whitman-leaves.txt']
Or	['whitman-leaves.txt']
d	['blake-poems.txt', 'whitman-leaves.txt']
loss	['milton-paradise.txt']
like	['chesterton-ball.txt']
lost	['chesterton-thursday.txt']
OF	['blake-poems.txt']
Profession	['shakespeare-caesar.txt']
laughing	['blake-poems.txt']
governess	['austen-emma.txt']
Milton	['milton-paradise.txt']
Vpon	['shakespeare-caesar.txt']
valleys	['blake-poems.txt']
old	['melville-moby_dick.txt', 'chesterton-thursday.txt']
obliged	['edgeworth-parents.txt']
proudly	['chesterton-thursday.txt']
somehow	['melville-moby_dick.txt']
Woodhouse	['austen-emma.txt']
Primus	['shakespeare-caesar.txt']
vse	['shakespeare-caesar.txt']
sight	['blake-poems.txt']
home	['shakespeare-caesar.txt', 'austen-emma.txt']
pale	['melville-moby_dick.txt']
Mary	['edgeworth-parents.txt']
Life	['chesterton-thursday.txt']
for	['chesterton-ball.txt', 'whitman-leaves.txt', 'austen-emma.txt']
1851	['melville-moby_dick.txt']
Men	['chesterton-thursday.txt']
1855	['whitman-leaves.txt']
everything	['chesterton-ball.txt']
pen	['blake-poems.txt']
Let	['bible-kjv.txt']
?	['shakespeare-caesar.txt']
He	['melville-moby_dick.txt']
regain	['milton-paradise.txt']
be	['chesterton-ball.txt', 'bible-kjv.txt']
School	['melville-moby_dick.txt']
who	['milton-paradise.txt', 'austen-emma.txt']
Eden	['milton-paradise.txt']
;"	['blake-poems.txt']
Near	['edgeworth-parents.txt']
by	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
First	['bible-kjv.txt']
wept	['blake-poems.txt']
on	['milton-paradise.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt']
sister	['austen-emma.txt']
of	['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Trade	['shakespeare-caesar.txt']
blessings	['austen-emma.txt']
or	['milton-paradise.txt', 'austen-emma.txt']
own	['chesterton-thursday.txt']
into	['milton-paradise.txt']
down	['blake-poems.txt']
weather	['chesterton-thursday.txt']
But	['shakespeare-caesar.txt']
remembrance	['austen-emma.txt']
Edgeworth	['edgeworth-parents.txt']
your	['shakespeare-caesar.txt']
Ball	['chesterton-ball.txt']
her	['edgeworth-parents.txt', 'austen-emma.txt']
Assistant	['edgeworth-parents.txt']
."	['blake-poems.txt']
there	['bible-kjv.txt', 'edgeworth-parents.txt']
long	['whitman-leaves.txt', 'edgeworth-parents.txt', 'austen-emma.txt']
Every	['blake-poems.txt']
caresses	['austen-emma.txt']
stars	['chesterton-ball.txt']
house	['austen-emma.txt']
:	['milton-paradise.txt', 'bible-kjv.txt', 'shakespeare-caesar.txt', 'chesterton-thursday.txt', 'blake-poems.txt']
was	['austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt']
Herman	['melville-moby_dick.txt']
happy	['blake-poems.txt', 'austen-emma.txt']
himself	['chesterton-ball.txt']
form	['bible-kjv.txt']
Science	['chesterton-thursday.txt']
but	['shakespeare-caesar.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt']
boys	['chesterton-thursday.txt']
Moby	['melville-moby_dick.txt']
reed	['blake-poems.txt']
Fla	['shakespeare-caesar.txt']
with	['milton-paradise.txt', 'austen-emma.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt']
mockingly	['melville-moby_dick.txt']
he	['bible-kjv.txt', 'blake-poems.txt']
unattempted	['milton-paradise.txt']
made	['blake-poems.txt']
up	['edgeworth-parents.txt']
us	['milton-paradise.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt']
Walt	['whitman-leaves.txt']
BOOK	['blake-poems.txt', 'whitman-leaves.txt']
faded	['chesterton-thursday.txt']
professor	['chesterton-ball.txt']
affection	['austen-emma.txt']
walke	['shakespeare-caesar.txt']
taste	['milton-paradise.txt']
moved	['bible-kjv.txt']
Cross	['chesterton-ball.txt']
Thursday	['chesterton-thursday.txt']
an	['austen-emma.txt']
Masse	['whitman-leaves.txt']
To	['chesterton-thursday.txt']
as	['edgeworth-parents.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt', 'austen-emma.txt']
clever	['austen-emma.txt']
at	['edgeworth-parents.txt']
Above	['milton-paradise.txt']
tumultuous	['whitman-leaves.txt']
some	['whitman-leaves.txt', 'austen-emma.txt']
ship	['chesterton-ball.txt']
again	['blake-poems.txt']
no	['chesterton-ball.txt', 'milton-paradise.txt']
when	['chesterton-thursday.txt']
other	['whitman-leaves.txt']
5	['bible-kjv.txt']
sick	['chesterton-thursday.txt']
Poems	['blake-poems.txt']
you	['shakespeare-caesar.txt', 'chesterton-thursday.txt']
Man	['milton-paradise.txt', 'chesterton-thursday.txt']
Like	['chesterton-thursday.txt']
Lost	['milton-paradise.txt']
separate	['whitman-leaves.txt']
Julius	['shakespeare-caesar.txt']
fell	['edgeworth-parents.txt']
Testament	['bible-kjv.txt']
Melville	['melville-moby_dick.txt']
ago	['austen-emma.txt']
together	['chesterton-thursday.txt']
Brought	['milton-paradise.txt']
As	['edgeworth-parents.txt']
write	['blake-poems.txt', 'whitman-leaves.txt']
oracle	['milton-paradise.txt']
rural	['blake-poems.txt']
once	['edgeworth-parents.txt']
songs	['blake-poems.txt']