1. 分块是用于实体识别的基本技术
示例:名词短语分块
#一个简单的基于正则表达式的NP分块的例子
sentence = [('the','DT'),('little','JJ'),('yellow','JJ'),('dog','NN'),\
('barked','VBD'),('at','IN'),('the','DT'),('cat','NN')]
grammer = "NP: {<DT>?<JJ>*<NN>}"
cp = nltk.RegexpParser(grammer)
result = cp.parse(sentence)
print(result)
(S
(NP the/DT little/JJ yellow/JJ dog/NN)
barked/VBD
at/IN
(NP the/DT cat/NN))
In[4]: result.draw()
2. 加缝器: 加缝器是从大块中去除标识符序列的过程
(1)如果匹配的标识符序列贯穿整块,那么这个整块将被去除
(2)如果标识符序列出现在块中间,那么这些标识符会被去除,在以前只有一个块的地方留下两个块
(3)如果序列在块的周边,这些标记会被去除,留下一个较小的块
In[8]: grammar= r""""NP:{<.*>+}
}<VBD|IN>+{
{<.*>+}"""
cp = nltk.RegexpParser(grammer)
print(cp.parse(sentence))
(S
(NP the/DT little/JJ yellow/JJ dog/NN)
barked/VBD
at/IN
(NP the/DT cat/NN))
3. 分块表示
#分块的表示
#使用最广泛的表示是IOB标记,I(inside,内部):块内的标识符子序列,
# B(begin,开始):分块的开始,O(outside,外部):所有其它的标识符
#开发和评估分块器
from nltk.corpus import conll2000
print(conll2000.chunked_sents('train.txt')[99])
print(conll2000.chunked_sents('train.txt',chunk_types=['NP'])[99])
(S
(PP Over/IN)
(NP a/DT cup/NN)
(PP of/IN)
(NP coffee/NN)
,/,
(NP Mr./NNP Stone/NNP)
(VP told/VBD)
(NP his/PRP$ story/NN)
./.)
(S
Over/IN
(NP a/DT cup/NN)
of/IN
(NP coffee/NN)
,/,
(NP Mr./NNP Stone/NNP)
told/VBD
(NP his/PRP$ story/NN)
./.)
使用正则规则能提升分块表现
#简单的评估和基准
cp = nltk.RegexpParser("")
test_sents = conll2000.chunked_sents('test.txt',chunk_types=['NP'])
print(cp.evaluate(test_sents))
ChunkParse score:
IOB Accuracy: 43.4%%
Precision: 0.0%%
Recall: 0.0%%
F-Measure: 0.0%%
grammer = r"NP: {<[CDJNP].*>+}"
cp = nltk.RegexpParser(grammer)
print(cp.evaluate(test_sents))
ChunkParse score:
IOB Accuracy: 87.7%%
Precision: 70.6%%
Recall: 67.8%%
F-Measure: 69.2%%
使用unigram标注器
#使用unigram标注器对名词短语分块
class UnigramChunker(nltk.ChunkParserI):
def __init__(self,train_sents):
train_data = [[(t,c) for w,t,c in nltk.chunk.tree2conlltags(sent)] for sent in train_sents]
self.tagger = nltk.UnigramTagger(train_data)
def parse(self,sentence):
pos_tags = [pos for (word,pos) in sentence]
tagged_pos_tags = self.tagger.tag(pos_tags)
chunktags = [chunktag for (pos,chunktag) in tagged_pos_tags]
conlltags = [(word,pos,chunktag) for ((word,pos),chunktag) in zip(sentence,chunktags)]
return nltk.chunk.conllstr2tree(conlltags)
test_sents = conll2000.chunked_sents('text.txt',chunk_types=["NP"])
train_sents = conll2000.chunked_sents('train.txt',chunk_types=["NP"])
unigram_chunker = UnigramChunker(train_sents)
print(unigram_chunker.evaluate(test_sents))
4. 关系抽取:注意原书代码nltk.sem.show_raw_rtuple(rel)会报错,这是因为show_raw_rtuple已经改为rtuple方法了
AttributeError: module 'nltk.sem' has no attribute 'show_raw_rtuple'
#方法之一是首先寻找所有(X,a,Y)形式的三元组,其中X,Y是指定类型的命名实体,a表示X和Y之间的关系的字符串。
#然后使用正则表达式从a的实体中抽出正在查找的关系。
import re
IN = re.compile(r'.*\bin\b(?!\b.+ing)')
for doc in nltk.corpus.ieer.parsed_docs('NYT_19980315'):
for rel in nltk.sem.extract_rels('ORG','LOC',doc,corpus='ieer',pattern=IN):
print(nltk.sem.rtuple(rel))
[ORG: 'WHYY'] 'in' [LOC: 'Philadelphia']
[ORG: 'McGlashan & Sarrail'] 'firm in' [LOC: 'San Mateo']
[ORG: 'Freedom Forum'] 'in' [LOC: 'Arlington']
[ORG: 'Brookings Institution'] ', the research group in' [LOC: 'Washington']
[ORG: 'Idealab'] ', a self-described business incubator based in' [LOC: 'Los Angeles']
[ORG: 'Open Text'] ', based in' [LOC: 'Waterloo']
[ORG: 'WGBH'] 'in' [LOC: 'Boston']
[ORG: 'Bastille Opera'] 'in' [LOC: 'Paris']
[ORG: 'Omnicom'] 'in' [LOC: 'New York']
[ORG: 'DDB Needham'] 'in' [LOC: 'New York']
[ORG: 'Kaplan Thaler Group'] 'in' [LOC: 'New York']
[ORG: 'BBDO South'] 'in' [LOC: 'Atlanta']
[ORG: 'Georgia-Pacific'] 'in' [LOC: 'Atlanta']