英语文本处理工具库——spaCy

spaCy的主要操作：

分词断句

import spacy
nlp = spacy.load('en')
doc = nlp('Hello World! My name is HanXiaoyang')
# 分词
for token in doc:
    print('"' + token.text + '"')
# 断句
for sent in doc.sents:
    print(sent)

每个token对象有着非常丰富的属性，如下的方式可以取出其中的部分属性。

doc = nlp("Next week I'll   be in Shanghai.")
for token in doc:
    print("{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}".format(
        token.text,
        token.idx,
        token.lemma_,
        token.is_punct,
        token.is_space,
        token.shape_,
        token.pos_,
        token.tag_
    ))
输出结果如下：
Next    0   next    False   False   Xxxx    ADJ JJ
week    5   week    False   False   xxxx    NOUN    NN
I   10  -PRON-  False   False   X   PRON    PRP
'll 11  will    False   False   'xx VERB    MD
    15      False   True        SPACE   _SP
be  17  be  False   False   xx  VERB    VB
in  20  in  False   False   xx  ADP IN
Shanghai    23  shanghai    False   False   Xxxxx   PROPN   NNP
.   31  .   True    False   .   PUNCT   .

词性标注

doc = nlp("Next week I'll be in Shanghai.")
print([(token.text, token.tag_) for token in doc])
输出结果：
[('Next', 'JJ'), ('week', 'NN'), ('I', 'PRP'), ("'ll", 'MD'), ('be', 'VB'), ('in', 'IN'), ('Shanghai', 'NNP'), ('.', '.')]

组块分析

spaCy可以自动检测名词短语，并输出根(root)词，比如下面的"Journal",“piece”,“currencies”

doc = nlp("Wall Street Journal just published an interesting piece on crypto currencies")
for chunk in doc.noun_chunks:
    print(chunk.text, chunk.label_, chunk.root.text)

输出结果：
Wall Street Journal NP Journal
an interesting piece NP piece
crypto currencies NP currencies

命名实体识别

doc = nlp("Two years ago, I lived in my Beijing.")
for ent in doc.ents:
    print(ent.text, ent.label_)

输出结果：
Two years ago DATE
BeijingGPE

还可以用非常漂亮的可视化做显示：

from spacy import displacy
displacy.render(doc, style='ent', jupyter=True)

句法依存分析

doc = nlp('Wall Street Journal just published an interesting piece on crypto currencies')
 
for token in doc:
    print("{0}/{1} <--{2}-- {3}/{4}".format(
        token.text, token.tag_, token.dep_, token.head.text, token.head.tag_))
输出结果：
Wall/NNP <--compound-- Street/NNP
Street/NNP <--compound-- Journal/NNP
Journal/NNP <--nsubj-- published/VBD
just/RB <--advmod-- published/VBD
published/VBD <--ROOT-- published/VBD
an/DT <--det-- piece/NN
interesting/JJ <--amod-- piece/NN
piece/NN <--dobj-- published/VBD
on/IN <--prep-- piece/NN
crypto/JJ <--compound-- currencies/NNS
currencies/NNS <--pobj-- on/IN

词向量

NLP中有一个非常强大的文本表示学习方法叫做word2vec，通过词的上下文学习到词语的稠密向量化表示，同时在这个表示形态下，语义相关的词在向量空间中会比较接近。也有类似v(爷爷)-v(奶奶) ≈ v(男人)-v(女人)的关系。
在spaCy中，要使用英文的词向量，需先下载预先训练好的结果。

python3 -m spacy download en_core_web_lg

词向量的应用：

nlp = spacy.load('en_core_web_lg')
from scipy import spatial

# 余弦相似度计算
cosine_similarity = lambda x, y: 1 - spatial.distance.cosine(x, y)

# 男人、女人、国王、女王 的词向量
man = nlp.vocab['man'].vector
woman = nlp.vocab['woman'].vector
queen = nlp.vocab['queen'].vector
king = nlp.vocab['king'].vector
 
# 我们对向量做一个简单的计算，"man" - "woman" + "queen"
maybe_king = man - woman + queen
computed_similarities = []

# 扫描整个词库的词向量做比对，召回最接近的词向量
for word in nlp.vocab:
    if not word.has_vector:
        continue
 
    similarity = cosine_similarity(maybe_king, word.vector)
    computed_similarities.append((word, similarity))

# 排序与最接近结果展示
computed_similarities = sorted(computed_similarities, key=lambda item: -item[1])
print([w[0].text for w in computed_similarities[:10]])

输出结果：
['Queen', 'QUEEN', 'queen', 'King', 'KING', 'king', 'KIng', 'Kings', 'KINGS', 'kings']

词汇与文本相似度

在词向量的基础上，spaCy提供了从词到文档的相似度计算的方法，下面的例子是它的使用方法。

# 词汇语义相似度(关联性)
banana = nlp.vocab['banana']
dog = nlp.vocab['dog']
fruit = nlp.vocab['fruit']
animal = nlp.vocab['animal']
 
print(dog.similarity(animal), dog.similarity(fruit)) # 0.6618534 0.23552845
print(banana.similarity(fruit), banana.similarity(animal)) # 0.67148364 0.2427285
# 文本语义相似度(关联性)
target = nlp("Cats are beautiful animals.")
 
doc1 = nlp("Dogs are awesome.")
doc2 = nlp("Some gorgeous creatures are felines.")
doc3 = nlp("Dolphins are swimming mammals.")
 
print(target.similarity(doc1))  # 0.8901765218466683
print(target.similarity(doc2))  # 0.9115828449161616
print(target.similarity(doc3))  # 0.7822956752876101