词向量--原理、word2vec、GloVe及其实现

这学期选了研究生的NLP课程，期间有几次homework，在此记录一下。第一次作业的题目是

Implement a word clustering method based on neural network language model.

• Use word2vec to train distributed representations;
• Analyze and visualize the (full or partial) result in nearest neighbors or linear structures;
• Optional: Use GloVe to train distributed representations and do the same analysis.

数据集

NLP的数据集有很多，这里我用了text8，它是enwiki8的前100,000,000个单词。里面只包含a-z以及空格，所以不用进行什么预处理。
用jupyter看一下text8的数据内容

text8 = open("./text8")
content = text8.readline()
content[:1000] 

OUTPUT

' anarchism originated as a term of abuse first used against early working class radicals including the diggers of the 
english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way 
to describe any act that used violent means to destroy the organization of society it has also been 
taken up as a positive label by self defined anarchists the word anarchism is derived from the greek without archons ruler 
chief king anarchism as a political philosophy is the belief that rulers are unnecessary and should be abolished although there are differing interpretations of what this means anarchism also refers to related social movements 
that advocate the elimination of authoritarian institutions particularly the state the word anarchy as most anarchists use 
it does not imply chaos nihilism or anomie but rather a harmonious anti authoritarian society in place of what are 
regarded as authoritarian political structures and coercive economic instituti'

word2vec

在word2vec方法出来以前，一般词向量都是one-hot形式。虽然它很简单，但是有两个缺点。一个是one-hot分裂了不同单词之间的联系；另一个就是它的表示太稀疏了。

分布式表示，也即词向量，可以解决上面这两个问题。它的基本思想是，将每个单词映射到一个向量上面。这个短的向量表示词的属性，也包含着与其他单词的关系。word2vec就是用的这个思想，它采用一个双层的神经网络进行词向量的学习。其有两个模型：CBOW(continuous bag-of-word)和SG(skip-gram)。

CBOW

CBOW是用上下文的单词预测中心单词：

考虑最简单的情况，只用一个周围词预测中心词。

假设单词表容量是$V$，词向量维度是$N$。网络输入的编码模式是one-hot向量，则其尺寸为$V\times 1$。第一层与第二层的网络权重为一个$V\times N$的矩阵：
$$W=\left(\begin{array}{ccc}{w}_{11}& \dots & w_{1N}\\ ⋮& \ddots & ⋮\\ w_{V1}& \dots & w_{VN}\end{array}\right)$$如果输入一个one-hot形式的向量$x={\left(0,0,\dots ,1,\dots ,0\right)}^T,x_I=0$到模型之中，那么第一层的对应的计算公式为
$$h=W^T{x}_I=W_{(I,\cdot )}{x}_I=v_I$$也就是说，矩阵$W$的第$I$行就是对应的单词表中的第$I$个单词。
下一层的权重是一个$N\times V$的矩阵 W ′ = { w i j ′ } W'=\{ w_{ij}' \} W′={wij′​}。用此矩阵可以计算最后一层的logits$$u_j=W^{\prime T}{h}_j=v_j^{\prime T}h$$之后就可以用softmax来得到单词$j$的后验表达式$$y_j=p(w_j|w_I)=\frac{e^{u_j}}{{\sum }_{k=1}^V{e}^{u_k}}=\frac{e^{v_j^{\prime T}h}}{{\sum }_{k=1}^V{e}^{v_j^{\prime T}h}}$$为了训练CBOW模型，我们需要定义一个损失函数。一个最直接的想法就是采用最大似然估计来最大化上述的后验概率$$\max p(w_o|w_I)$$也就是$$\begin{array}{rl}{E}_o& =-\log p(w_o|w_I)\\ & =-\log \frac{e^{u_o}}{{\sum }_{k=1}^V{e}^{u_k}}\\ & =-u_o+\sum _{k=1}^V\log e^{u_k}\end{array}$$接着我们计算后向传导。首先定义$$\frac{\partial E_o}{\partial u_j}=\frac{e^{u_j}}{{\sum }_{k=1}^V{e}^{u_k}}-t_j=y_j-t_j$$其中$t_j=\{\begin{array}{rl}1& \text{if }j=o\\ 0& \text{otherwise}\end{array}$接着通过链式法则推导梯度$$\begin{array}{c}\frac{\partial E_o}{\partial v_j}=\frac{\partial E_o}{\partial u_j}\cdot \frac{\partial u_j}{\partial v_j}=(y_j-t_j)\cdot h\\ \frac{\partial E_o}{\partial v_I}=\sum _{j=1}^V\frac{\partial E_o}{\partial u_j}\cdot \frac{\partial u_j}{\partial h_j}\cdot \frac{\partial h_j}{\partial v_I}=\sum _{j=1}^V(y_j-t_j)v_j^{\prime }\end{array}$$终于将其推倒完成了。需要注意的是，真正实现的时候使用多个周边词预测中心词，此时输入向量的维度将变为$C\times V$。

表达式是类似的，在计算中间的$h$向量时，用这$C$个输入的平均值代替$$h=\frac{1}{C}{W}^T(x_1+\cdots +x_C)=\frac{1}{C}(v_1+\cdots v_C)$$损失函数变为$$\begin{array}{rl}E& =-\log p(w_o|w_{I,1},\cdots ,w_{I,C})\\ & =-u_o+\sum _{k=1}^V\log e^{u_k}\\ & =-v_o^{\prime T}+\sum _{k=1}^V\log e^{v_{k}h}\end{array}$$

Skip-Gram模型

SG模型与上述的CBOW类似，唯一不同的是使用中心词预测周围词。


损失函数变为$$\begin{array}{rl}E& =-\log p(w_{o,1},\cdots ,w_{o,C}|w_I)\\ & =-\log \left(\prod _{c=1}^C\frac{u_{j_c}}{{\sum }_{k=1}^V{e}^{u_k}}\right)\\ & =-\sum _{c=1}^C{u}_{j_c}+C\sum _{k=1}^V\log u_k\end{array}$$

改进的方法

层次softmax

层次softmax基于霍夫曼树。这使我们能够将计算一个单词的概率分解为一系列概率计算，从而使我们不必计算所有单词的昂贵归一化。

负采样

负采样思想是基于噪声对比估计（类似的对抗生成对抗网络）的概念，该概念坚持认为，好的模型应通过逻辑回归将伪信号与真实信号区分开。
同样，负采样目标背后的动机类似于随机梯度下降：与其考虑到我们拥有的数千个观测值，每次都改变所有权重，我们仅使用其中的$K$个并增加计算量效率也显着提高（取决于阴性样品的数量）。
在原始论文中，作者使用会标分布选择否定词。 选择一个单词作为否定采样的概率与其频率有关，经验公式为$$P(w_i)=\frac{f(w_i{)}^{\frac{3}{4}}}{{\sum }_{j=0}^{n}f(w_j{)}^{\frac{3}{4}}}$$

代码实现

这部分使用gensim软件包进行实现。首先导入所需包

from gensim.models import word2vec

加载语料

sentences = word2vec.Text8Corpus("./text8")

gensim提供了很友好的API，word2vec.Word2Vec()函数的原型如下所示

class gensim.models.word2vec.Word2Vec(sentences=None, corpus_file=None, size=100, alpha=0.025, window=5, 
min_count=5, max_vocab_size=None, sample=0.001, seed=1, workers=3, min_alpha=0.0001, sg=0, hs=0, negative=5, ns_exponent=0.75, cbow_mean=1, hashfxn=<built-in function hash>, iter=5, null_word=0, trim_rule=None,
 sorted_vocab=1, batch_words=10000, compute_loss=False, callbacks=(), max_final_vocab=None)

这里设置词向量的维度为200

model = word2vec.Word2Vec(sentences, size=200)

训练完后看一下其中某一向量

vec_king = model.wv["king"]
vec_king 

OUTPUT

array([-2.33487463e+00, -2.94242352e-01,  8.58403683e-01,  8.11171889e-01,3.38901699e-01, -2.57826972e+00, 
-1.46598172e+00,  2.26720309e+00,...
-6.03341758e-01, -2.41992116e+00, 1.58592343e+00, 
1.19147336e+00], dtype=float32)

当然也可以保存模型以及加载

model.save("./text8.model")
model1 = word2vec.Word2Vec.load('text8.model')
model1

OUTPUT

<gensim.models.word2vec.Word2Vec at 0x156967860>

词向量最有名的应用莫过于从语义角度解释词的差异

for e in model.most_similar("man"): print(e[0], e[1])

OUTPUT

woman 0.6996182203292847
girl 0.5773996710777283
boy 0.5519055128097534
creature 0.5370252728462219
person 0.5368808507919312
gentleman 0.5074084997177124
stranger 0.503169059753418
men 0.4922099709510803
evil 0.4899784326553345
mortal 0.48563891649246216

意思是说和man最接近的单词是woman，余弦相似度为0.699。
使用t-SNE算法可以对词向量进行二维投影分析：

from sklearn.decomposition import IncrementalPCA    
from sklearn.manifold import TSNE                  
import numpy as np                                 

def reduce_dimensions(model):
num_dimensions = 2  

vectors = [] 
    labels = [] 
    cnt = 0
    for word in model.wv.vocab:
        cnt += 1
        vectors.append(model.wv[word])
        labels.append(word)
        if cnt == 10000:
            break

    vectors = np.asarray(vectors)
    labels = np.asarray(labels)

    vectors = np.asarray(vectors)
    tsne = TSNE(n_components=num_dimensions, random_state=0)
    vectors = tsne.fit_transform(vectors)

    x_vals = [v[0] for v in vectors]
    y_vals = [v[1] for v in vectors]
    return x_vals, y_vals, labels

x_vals, y_vals, labels = reduce_dimensions(model)

def plot_with_matplotlib(x_vals, y_vals, labels):
    import matplotlib.pyplot as plt
    import random

    random.seed(0)

    plt.figure(figsize=(12, 12))
    plt.scatter(x_vals, y_vals)

    indices = list(range(len(labels)))
    selected_indices = random.sample(indices, 25)
    for i in selected_indices:
        plt.annotate(labels[i], (x_vals[i], y_vals[i]))

plot_with_matplotlib(x_vals, y_vals, labels)

OUTPUT

GloVe

Standford的GloVe包是基于C和shell的

./demo.sh -CORPUS text8 -VECTOR_SIZE 200 -WINDOW_SIZE 5

OUTPUT

mkdir -p build

$ build/vocab_count -min-count 5 -verbose 2 < text8 > vocab.txt
BUILDING VOCABULARY
Processed 17005207 tokens.
Counted 253854 unique words.
Truncating vocabulary at min count 5.
Using vocabulary of size 71290.

...
TRAINING MODEL
Read 60666468 lines.
Initializing parameters...Using random seed 1591149439
done.
vector size: 50
vocab size: 71290
x_max: 10.000000
alpha: 0.750000
06/03/20 - 11:10.10AM, iter: 001, cost: 0.071225
06/03/20 - 11:10.19AM, iter: 002, cost: 0.052664
06/03/20 - 11:10.28AM, iter: 003, cost: 0.046694
...
06/03/20 - 11:12.07AM, iter: 014, cost: 0.036438
06/03/20 - 11:12.16AM, iter: 015, cost: 0.036235

训练好的模型可以用gensim加载

from gensim.test.utils import datapath, get_tmpfile
from gensim.models import KeyedVectors

glove_file = './glove/vectors.txt'
tmp_file = './glove2word2vector.txt'

from gensim.scripts.glove2word2vec import glove2word2vec
glove2word2vec(glove_file, tmp_file)

model = KeyedVectors.load_word2vec_format(tmp_file)
for e in model.most_similar("man"): print(e[0], e[1])

OUTPUT

woman 0.8029251098632812
girl 0.7217116355895996
my 0.714549720287323
thing 0.6998955011367798
love 0.6971036791801453
said 0.6951542496681213
young 0.6937921047210693
who 0.6918587684631348
she 0.690385639667511
story 0.688470721244812

可以看到和word2vec相比，GloVe对词向量的计算似乎更好一些。

Refenrence

[1] https://zhuanlan.zhihu.com/p/50125315
[2] https://blog.csdn.net/qq_41664845/article/details/82971728
[3] https://zhuanlan.zhihu.com/p/136247620
[4] https://ruder.io/word-embeddings-softmax/index.html#hierarchicalsoftmax
[5] https://www.youtube.com/watch?v=B95LTf2rVWM
[6] https://towardsdatascience.com/hierarchical-softmax-and-negative-sampling-short-notes-worth-telling-2672010dbe08
[7]https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#sphx-glr-auto-examples-tutorials-run-word2vec-py
[8] https://blog.csdn.net/hustqb/article/details/78144384
[9]https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#sphx-glr-auto-examples-tutorials-run-word2vec-py
[10] https://zhuanlan.zhihu.com/p/31023929